Release v1.6.64

RELEASE.md

@@ -1,38 +1,31 @@
-# v1.6.63 — Driver, customer and contact accounts stay out of the console
+# v1.6.64 — Order reporting and test SMS with the entered credentials
 
-## Improvements
+## Improvements for reporting
 
-- Treat `driver`, `customer` and `contact` users as managed accounts: the FleetOps profile owns them, not IAM. `User` gains `MANAGED_TYPES`, `isManagedAccount()`, `isStaffAccount()`, `canAccessConsole()`, `canHoldConsoleSession()` and a `managed()` scope.
-- Promote instead of duplicating. When IAM creates or invites a team member whose email or phone belongs to a managed account in the organization, that account becomes a `user`. It gets the chosen role, permissions and policies and a join invite, and keeps its driver and customer profiles. The response carries `promoted_from`. Accepting any IAM invite also promotes a managed account and asks it to set a console password.
-
-## Improvements for IAM
-
-- Let IAM admins ask a user to verify their email or phone. `POST users/{id}/send-verification` sends a one-click link by email or SMS; it lasts 48 hours. The public `auth/confirm-contact-verification` confirms it without signing in, and refuses the link if the address changed since. `users/verify/{id}` takes a `channel` (email by default, or phone). `UserFilter` adds `email_verified`, `phone_verified`, `country` and `timezone` for the new IAM columns.
+- Declare row-level expression columns with `Column::expression($name, $sql, $type)`. Bare names resolve against the table or relationship that declares it, so `JSON_EXTRACT(meta, '$.quantity')` on `payload.entities` reads the joined entity's `meta`. An expression column can be selected, filtered, sorted, grouped by and aggregated.
+- Summary columns (`Column::count/sum/avg`) are flagged `aggregate` and resolve to their computation. Without grouping they return a single summary row; with grouping they sit beside the group keys.
+- `Table::softDeletes()` and `Relationship::softDeletes()` leave out soft-deleted rows. On joins the filter goes in the `ON` clause, so LEFT joins keep the parent row.
+- Computed columns can be group keys, conditions and sort columns, and a grouped report can be sorted by an aggregate's alias. A new `count_distinct` aggregate is available.
+- Custom expressions accept the JSON functions, `DATE()`, `CAST(… AS DECIMAL(15,2))` and the other cast types, `DISTINCT`, `IN`, `GROUP_CONCAT(… ORDER BY … SEPARATOR …)`, `->`/`->>` and `INTERVAL n UNIT`.
+- `public_id` and `internal_id` are no longer hidden as foreign keys, so ID columns appear in the column picker.
+- Relationship columns are labelled with the whole relationship name ("Order Config Namespace", not "Order Namespace"), and aggregate labels use the column label ("Sum (Quantity)").
+- `_key` and `_import_id` are never listed or selectable, whatever a schema declares.
 
 ## Fixes
 
-- Keep managed accounts out of the console:
-  - `auth/login` refuses drivers, contacts and customers. Customers keep the `customer_login_not_allowed` code; drivers and contacts get `console_access_not_allowed`.
-  - Session restore, bootstrap, 2FA verification, verify-email tokens and impersonation refuse drivers and contacts.
-  - Customers are still allowed on those endpoints because the customer portal runs inside the console and restores its session through them.
-- Free a deleted user's email and phone so a new account can use them. On soft delete they move to `meta.deleted_identity`; restoring the user puts them back only if no other account has taken them.
+- Test SMS Provider and Test Twilio in Admin › System Config › Services use the credentials entered in the form (fleetbase/fleetbase#680). Under Octane the Twilio client was built once per worker, so a test failed with "Credentials are required to create a Client" or reported success for the saved account. The endpoints now rebuild the client from the request's config and release it after the send.
 
 ## Security
 
-- Never grant the Administrator role by default. Before this fix:
-  - Creating or inviting a user without a role gave them the Administrator role, and so full organization access. Reported for IAM › Customers › Add customer with a blank Role.
-  - Accepting an invite with no role also granted it, and `joinOrganization` ignored the invite's role altogether.
-
-  Now:
-  - A role is required when creating or inviting a user; without one the request returns 422.
-  - `Company::addUser`, `Company::assignUser` and `User::assignCompany` assign no role unless one is given.
-  - An invite without a role joins with no role.
-- Only admins or holders of the Administrator role may grant the Administrator role (403 otherwise), on create, invite and role update.
+- Every computed column is validated up front, including in grouped reports. Names must be safe identifiers, and `SELECT` is forbidden.
+- Schema-declared columns always take their SQL from the registry, never from the request. Group keys, aggregate columns, sort columns and condition fields must be allowed or computed columns.
+- Sort direction is normalised to `asc`/`desc`, and grouped reports validate `aggregateBy.computation`.
 
-## Reliability
+## Behaviour changes
 
-- Cover the console guards for each account type, identity release and restore, and promotion through create, invite and invite acceptance.
+- In a grouped report, a selected column that is neither a group key nor aggregated is now an error instead of being dropped.
+- Invalid report shapes fail with a clear message instead of an SQL error.
 
-A database migration is not required. No configuration change is needed. The FleetOps side ships in fleetbase/fleetops#338.
+A database migration is not required. No configuration change is needed. The FleetOps order report schema ships in fleetbase/fleetops v0.6.70, and the report builder changes in fleetbase/ember-ui v0.4.4.
 
-Changes: [#264](https://github.com/fleetbase/core-api/pull/264), [#266](https://github.com/fleetbase/core-api/pull/266), [#267](https://github.com/fleetbase/core-api/pull/267).
+Changes: [#269](https://github.com/fleetbase/core-api/pull/269), [#270](https://github.com/fleetbase/core-api/pull/270).

composer.json

@@ -1,6 +1,6 @@
 {
     "name": "fleetbase/core-api",
-    "version": "1.6.63",
+    "version": "1.6.64",
     "description": "Core Framework and Resources for Fleetbase API",
     "keywords": [
         "fleetbase",

src/Http/Controllers/Internal/v1/SettingController.php

@@ -21,6 +21,9 @@
 use Fleetbase\Services\SmsService;
 use Fleetbase\Support\PlatformApi;
 use Fleetbase\Support\Utils;
+use Fleetbase\Twilio\Manager as TwilioManager;
+use Fleetbase\Twilio\Support\Laravel\Facade as TwilioFacade;
+use Fleetbase\Twilio\TwilioInterface;
 use Illuminate\Http\Request;
 use Illuminate\Notifications\AnonymousNotifiable;
 use Illuminate\Support\Arr;
@@ -594,6 +597,8 @@ public function testSmsProviderConfig(AdminRequest $request)
         } catch (\Throwable $e) {
             $responseMessage = $e->getMessage();
             $status          = 'error';
+        } finally {
+            $this->releaseTwilioClient();
         }
 
         return response()->json([
@@ -843,12 +848,13 @@ public function testTwilioConfig(AdminRequest $request)
 
         // Set config from request
         config(['twilio.twilio.connections.twilio.sid' => $sid, 'twilio.twilio.connections.twilio.token' => $token, 'twilio.twilio.connections.twilio.from' => $from]);
+        $this->refreshTwilioClient();
 
         $message = 'Twilio configuration is successful, SMS sent to ' . $phone . '.';
         $status  = 'success';
 
         try {
-            \Fleetbase\Twilio\Support\Laravel\Facade::message($phone, 'This is a Twilio test from Fleetbase');
+            TwilioFacade::message($phone, 'This is a Twilio test from Fleetbase');
         } catch (\Twilio\Exceptions\RestException $e) {
             $message = $e->getMessage();
             $status  = 'error';
@@ -861,6 +867,8 @@ public function testTwilioConfig(AdminRequest $request)
         } catch (\Error $e) {
             $message = $e->getMessage();
             $status  = 'error';
+        } finally {
+            $this->releaseTwilioClient();
         }
 
         return response()->json(['status' => $status, 'message' => $message]);
@@ -899,6 +907,7 @@ protected function setTemporarySmsProviderConfig(string $provider, array $provid
                 'services.twilio'                  => array_replace_recursive(config('services.twilio', []), $providerConfig),
                 'twilio.twilio.connections.twilio' => array_replace_recursive(config('twilio.twilio.connections.twilio', []), $providerConfig),
             ]);
+            $this->refreshTwilioClient();
         }
 
         if ($provider === SmsService::PROVIDER_CALLPRO) {
@@ -908,6 +917,49 @@ protected function setTemporarySmsProviderConfig(string $provider, array $provid
         }
     }
 
+    /**
+     * Rebuild the Twilio client from the config just applied.
+     *
+     * The Twilio manager copies its connection settings when it is built, and both the
+     * container singleton and the facade's static cache keep the built manager. Under
+     * Octane the facade cache outlives the request, so a test send kept using the
+     * credentials the worker first saw: it failed with "Credentials are required to create
+     * a Client" when none were saved, or quietly used the saved ones instead of those just
+     * entered. A stand-in bound in place of the real manager is left alone.
+     */
+    protected function refreshTwilioClient(): void
+    {
+        TwilioFacade::clearResolvedInstance('twilio');
+
+        if (!app()->bound('twilio')) {
+            return;
+        }
+
+        $current = app()->resolved('twilio') ? app('twilio') : null;
+        if ($current !== null && !($current instanceof TwilioManager)) {
+            return;
+        }
+
+        // Build it here from this request's config: the provider's singleton closure reads
+        // the config of the application it was registered on, which under Octane is the
+        // worker's base application, not the copy this request just changed.
+        $manager = $current ? get_class($current) : TwilioManager::class;
+        $config  = config('twilio.twilio', []);
+
+        app()->instance('twilio', new $manager($config['default'] ?? 'twilio', $config['connections'] ?? []));
+        app()->forgetInstance(TwilioInterface::class);
+    }
+
+    /**
+     * Forget the facade's cached Twilio client once a test send is done, so the credentials
+     * under test don't outlive this request in a long-running worker and get used for real
+     * messages (verification codes, notifications) sent by later requests.
+     */
+    protected function releaseTwilioClient(): void
+    {
+        TwilioFacade::clearResolvedInstance('twilio');
+    }
+
     /**
      * Sends a test exception to Sentry.
      *

src/Support/Reporting/ComputedColumnValidator.php

@@ -6,6 +6,28 @@
 
 class ComputedColumnValidator
 {
+    /**
+     * Keywords that are never column references.
+     */
+    public const SQL_KEYWORDS = [
+        'INTERVAL', 'AND', 'OR', 'XOR', 'NOT', 'IS', 'NULL', 'TRUE', 'FALSE',
+        'AS', 'FROM', 'WHERE', 'DIV', 'CASE', 'WHEN', 'THEN', 'ELSE', 'END',
+        'DISTINCT', 'IN', 'LIKE', 'BETWEEN', 'ESCAPE', 'REGEXP',
+        'ORDER', 'BY', 'ASC', 'DESC', 'SEPARATOR',
+    ];
+
+    /**
+     * Cast types and interval units. These are keywords after `AS` / `INTERVAL <n>` (e.g.
+     * `CAST(x AS SIGNED)`, `INTERVAL 7 DAY`) but may also be real column names (e.g. `time`).
+     */
+    public const CONTEXTUAL_KEYWORDS = [
+        // CAST / CONVERT target types
+        'DECIMAL', 'SIGNED', 'UNSIGNED', 'INTEGER', 'INT', 'CHAR', 'NCHAR', 'BINARY',
+        'DATE', 'DATETIME', 'TIME', 'DOUBLE', 'FLOAT', 'REAL', 'JSON',
+        // INTERVAL units
+        'MICROSECOND', 'SECOND', 'MINUTE', 'HOUR', 'DAY', 'WEEK', 'MONTH', 'QUARTER', 'YEAR',
+    ];
+
     /**
      * Allowed SQL functions.
      */
@@ -25,6 +47,8 @@ class ComputedColumnValidator
         'SECOND',
         'DATE_FORMAT',
         'LAST_DAY',           // Get last day of month
+        'DATE',               // Date part of a datetime, e.g. DATE(created_at)
+        'TIME',               // Time part of a datetime
         'DAYOFWEEK',          // Get day of week (1=Sunday, 7=Saturday)
         'DAYOFMONTH',         // Get day of month (1-31)
         'DAYOFYEAR',          // Get day of year (1-366)
@@ -118,6 +142,23 @@ class ComputedColumnValidator
         // Type Conversion
         'CAST',
         'CONVERT',
+        'DECIMAL',            // CAST(x AS DECIMAL(10,2))
+        'CHAR',               // CAST(x AS CHAR(20))
+        'BINARY',
+        'DOUBLE',
+        'FLOAT',
+
+        // JSON Functions, e.g. JSON_UNQUOTE(JSON_EXTRACT(meta, '$.total'))
+        'JSON_EXTRACT',
+        'JSON_UNQUOTE',
+        'JSON_VALUE',
+        'JSON_LENGTH',
+        'JSON_CONTAINS',
+        'JSON_CONTAINS_PATH',
+        'JSON_KEYS',
+        'JSON_TYPE',
+        'JSON_VALID',
+        'JSON_SEARCH',
 
         // Other Utility Functions
         'INTERVAL',           // For date arithmetic
@@ -141,7 +182,7 @@ class ComputedColumnValidator
         'ALTER', 'CREATE', 'GRANT', 'REVOKE',
         'EXEC', 'EXECUTE', 'UNION', 'INTO',
         'INFORMATION_SCHEMA', 'LOAD_FILE', 'OUTFILE',
-        'DUMPFILE', 'BENCHMARK', 'SLEEP',
+        'DUMPFILE', 'BENCHMARK', 'SLEEP', 'SELECT',
     ];
 
     protected ReportSchemaRegistry $registry;
@@ -221,7 +262,7 @@ protected function validateFunctions(string $expression): array
         $errors = [];
 
         // Match function calls: FUNCTION_NAME(
-        preg_match_all('/([A-Z_]+)\s*\(/i', $expression, $matches);
+        preg_match_all('/\b([A-Z_][A-Z0-9_]*)\s*\(/i', $this->removeStringLiterals($expression), $matches);
 
         if (!empty($matches[1])) {
             foreach ($matches[1] as $function) {
@@ -342,10 +383,7 @@ protected function removeStringLiterals(string $expression): string
      */
     protected function isKeywordOrLiteral(string $word): bool
     {
-        $keywords = array_merge($this->allowedFunctions, $this->allowedOperators, [
-            'TRUE', 'FALSE', 'NULL', 'AS', 'FROM', 'WHERE',
-            'INTERVAL', 'DAY', 'MONTH', 'YEAR', 'HOUR', 'MINUTE', 'SECOND',
-        ]);
+        $keywords = array_merge($this->allowedFunctions, $this->allowedOperators, static::SQL_KEYWORDS, static::CONTEXTUAL_KEYWORDS);
 
         return in_array(strtoupper($word), $keywords) || is_numeric($word);
     }
@@ -393,7 +431,7 @@ protected function isValidColumnReference(string $columnRef, Table $table): bool
      */
     protected function columnExistsInTable(string $columnName, Table $table): bool
     {
-        $columns = $table->getColumns();
+        $columns = $table->getAllColumns();
 
         foreach ($columns as $column) {
             if ($column->getName() === $columnName) {