Skip to content

Web UI

External libraries will not be documented here. View their official docs:

:::{note} All file paths in this page are relative to the root of the Web UI project (perseus-v2/software/web-ui). :::

Server

SvelteKit provides several methods of assisting with server communication, however I would recommend using the node server and WebSockets to prevent weird bugs from components re-rendering. Since Svelte doesn't provide a native interface for creating custom node servers an injected server is used. When writing modules for the WebSocket connection, keep everything inside the src/server/scripts folder and from there export the functions that handle WebSocket messages. This allows code to be called in the vite plugin inside vite.config.ts for testing and web-ui/src/server/server.js for production.

Widgets

Creation Script

To assist with creating widgets there is a basic script that populates a file with all the required boilerplate.

Usage

The script is located in the Web UI directory root. When calling the script provide the name of the file with no spaces as an argument:

./create-widget.sh <file-name>

:::{note} The filename is not the name of the widget which is set to "New widget" by default. :::

Widget Structure

The template generated by the script contains everything that a component needs to be considered a widget. Additionally, some optional fields have been added but commented out. The template is shown below:

<script lang="ts" module>
  // This is to expose the widget settings to the panel. Code in here will only run once when the widget is first loaded.
  import type {
    WidgetGroupType,
    WidgetSettingsType,
  } from "$lib/scripts/state.svelte";

  export const name = "New Widget";
  // These properties are optional
  // export const description = 'Description of the widget goes here';
  // export const group: WidgetGroupType = 'Group Name';
  // export const isRosDependent = true; // Set to true if the widget requires a ROS connection

  export const settings: WidgetSettingsType = $state<WidgetSettingsType>({
    groups: {},
  });
</script>

<script lang="ts">
  // import { getRosConnection } from '$lib/scripts/ros-bridge.svelte'; // ROSLIBJS docs here: https://robotwebtools.github.io/roslibjs/Service.html
  // import * as ROSLIB from 'roslib';

  // Widget logic goes here
</script>

<p>New component</p>

Exports

The first script tag has the module property meaning it is run once on widget initialisation and it also allows for exports. The values exported from here are used to define widget properties. The available properties are: | Name | Type | Description | |------|------|-------------| | name | string | This is where you can set the name of the widget which will be displayed in the Web UI. This is also used as a unique key and duplicate widgets will not be loaded. | | description | string (Optional) | Provide a description for the widget which can be seen in the settings panel. While there is no character limit, this should be kept to 1 to 2 sentences. | | group | "ROS" \| "CAN Bus" \| "Gstreamer" \| "Misc" (Optional) | This sets the group that the widget is part of in the "add widget" dropdown. If new groups are needed edit the type located in src/lib/scripts/state.svelte.ts. | | isRosDependant | boolean (Optional) | This should be set to true for any widget that uses the getRosConnection() function. This will make any widgets lock when ROS2 connection is lost. | | settings | WidgetSettingsType (More details below) | This object is evaluated into the settings panel and is saved as a JSON string in the layout database. |

Settings

The settings menu is designed to allow widgets to persist their state between sessions.

The type used is defined as:

export interface WidgetSettingsType {
  groups: Record<
    string,
    Record<
      string,
      {
        type: "text" | "number" | "select" | "switch" | "button" | "readonly";
        description?: string;
        value?: string;
        options?: { value: string; label: string }[]; // these are not saved as options are typically session dependent
        disabled?: boolean;
        action?: () => string | null;
      }
    >
  >;
}

The first Record represents the collapsible setting groups and the nested Record represents the setting fields. The fields types available are:

Text

Creates a text input field.

{
    type: "text",
    value?: string,
    description?: string,
    disabled?: boolean
}
Number

Creates a number input field with increment and decrement buttons (Options are the same as Text except with type: number).

Select

Drop down menu which should always have the options value set.

{
    type: "select",
    value?: string,
    options?: Array<{
        value: string,
        label: string
    }>,
    description?: string,
    disabled?: boolean
}

:::{note} The options field is not stored in the database as this is typically static data. :::

Switch

Boolean switch input (This field also has the same properties as Text).

Button

Buttons should always have an associated action. Their action cannot be saved meaning any buttons that are dynamically added to the widget state must also have their action set on mount.

{
    type: "text",
    description?: string,
    disabled?: boolean,
    action?: () => string | null
}

If the action returns a string, it will be displayed in a toast. Additionally, the button will be disables if the action is not set.

Readonly

This is for saving any additional configuration that doesnt make sense for users to have direct access to. This field is the same as a Text field except the disabled field has no effect.

:::{note} All keys used in this object are converted to sentence case and used as labels to avoid needing extra fields. :::

Logic

The second script tag is where all your TypeScript code should go. The commented imports in this script are added for convenience as they are needed for any ROS2 related widget and can be removed otherwise. To listen for updates in the settings object, wrap references to it in either an $effect or $derived rune.

:::{tip} If the setting is referenced multiple times or needs a type conversion, to improve readability, use a $derived rune to alias it:

let mySetting = $derived<number>(
  Number(settings.groups.general.numberInput.value),
);

:::

Markdown

The remainder of the file is where mark down goes. Tailwind classes can be used on widgets for styling or a <style></style> block, although the former is strongly preferred if possible.

Widget manager internals

Internally the widget manager is made up of the following components:

  • The custom widgets: /src/lib/widgets/
  • Widget state script: /src/lib/scripts/state.svelte.ts
  • Widget canvas: /src/lib/components/widget-canvas.svelte
  • Widget wrapper: /src/lib/components/widget.svelte

Custom Widget Components

All the files in the top level of the /src/lib/widgets/ directory are loaded as widgets by the server script in +layout.svelte before being converted to fit the widget type defined as:

export interface WidgetType {
  name: string;
  description?: string;
  group?: WidgetGroupType;
  isRosDependent?: boolean;
  component: Component;
  settings: WidgetSettingsType;
  layoutProps?: {
    x: number;
    y: number;
    w: number;
    h: number;
  };
}

:::{important} Any extra components or scripts needed for widgets should go in a folder sharing the same name as the widget in the src/lib/widgets directory. For example:

src/lib/widgets
├── myWidget
│       ├──myComponent.svelte
│       └── myScript.ts
└── myWidget.svelte

:::

Widget State

This file holds the reactive state of the layouts and widgets. The main function it exports is getWidgetsByLayoutId which is used to get a list of the widgets currently in use. Additionally, since the widget's persistent state is stored in the layout database, it uses the loaded widgets as a template and populates them using the saved state data.

Widget Canvas

The widget canvas is responsible for the drag and drop functionality of the Web UI. The drag and drop engine is svelte-grid-extended which uses svelte 4.2. While this library works, future releases of Svelte might release breaking changes. In the event of breaking changes, this library will need to be ported, rewritten, or replaced. Each widget has its location saved to the database on every movement. Using the liveQuery API from Remult all of these changes are automatically loaded by other clients.

Widget Wrapper

This component does two things, create an instance of the custom widget component and generate the widget settings menu from the WidgetSettingsType instance.

WebRTC Videos

Camera server

The function of the camera server is to:

  • Provide an interface to connected linux video devices
  • Generate Gstreamer pipelines and manage instances
  • Communicate with the main web server

Linux device interface: In the folder: /dev/v4l/by-id/ each camera has its ID listed. A node file watcher is used to get a live list of devices in this folder allowing for connect and disconnect handling. :::{note} Each video device has 2 files associated with it. The one with the suffix -index0 is the correct one and all other only contain meta data about the camera. :::

Gstreamer Pipelines

Each Gstreamer instance is tracked with the videos resolution and transform so the server known when a restart is necessary. Each pipeline uses these plugins: webrtcsink, v4l2src, videoflip, videoconvert to apply all the required processing as seen in this example pipeline:

gst-launch1.0 webrtcsink stun-server=NULL name=ws signaller::uri="ws://{web server host}:8443" meta="meta,device={device name}" v4l2src device=/dev/v4l/by-id/{device name} ! video/x-raw,width=320,height=240 ! videoconvert ! videoconvert method=none ! ws.

:::{note} The signaller::uri="ws://hostname:8443" property sets the signalling server to location as it is run by the main web server. Furthermore, the meta property assigns meta data to each WebRTC stream allowing the stream origin to be determined. :::

:::{tip} To fix a permission denied error when trying to connect to a camera use:

sudo chmod 777 /dev/video*

This gives read/write/execute permissions to all video devices. :::

WebSocket Communication Protocol

The communication with the web server is conducted via a WebSocket with the goal of sending information about the available devices to the client and then initialising requested streams. All messages are sent using the following interface.

interface CameraEventType {
  type: "camera";
  action:
    | "group-description"
    | "kill"
    | "request-groups"
    | "request-stream"
    | "group-terminated"
    | "device-disconnect";
  data: {
    devices?: string[];
    resolution?: { width: number; height: number };
    transform?: videoTransformType;
    forceRestart?: boolean;
  };
}

ROS2 Interface

Connection Handling

The WebSocket connection is managed inside the file: src/lib/scripts/ros-bridge.svelte.ts. The exports of this file are the functions: getRosConnection, connectRos and disconnectRos.

The connect and disconnect functions are used by the connection manager at the top middle of the Web UI layout and should only be used internally.

getRosConnection should be used to access the current ROS2 connection instance. This function returns a $state rune that is either false or the handle for the ROS connection. This allows for very clear boolean evaluation in if statements and integration with the Svelte reactivity suite.

Development Pattern

All ROS2 dependent widgets should have either an $effect or $derived.by rune ($effect is recommended as most use cases will not have a return value) that follows a pattern similar to:

$effect(() => {
  const ros = getRosConnection();

  if (ros) {
    // initialise ROS2 variables
  }
});

As a result of this effect pattern the following rules should be followed while developing:

  • Avoid making ROS2 related variables reactive as they will be assigned inside an $effect and that will cause a render loop.
  • Do not initialise anything related to ROS2 in the onMount hook as there is no guarantee that a ROS2 connection is available and will require duplicate code.
  • Always set isRosDependant to true and group to 'ROS' if using the getRosConnection() function (When using ROS2 purely to access CAN data the group export should be "CAN Bus").
  • Dispose of all ROS2 resources you created in the function returned by the onMount hook (the return value is run on unmount).

CAN Bus Interface

No direct interface between the Web UI and CAN exists. Instead the work around is to read CAN Bus data with a ROS2 node and publish on a ROS2 topic.

Topics used for relaying CAN Bus data should favour lower frequencies to reduce network load. If a node already consumes the required CAN Bus data then the topic should be created in there otherwise a new node inside software/ros_ws/src/perseus_can_if should be created. Additionally, these topics should be prefixed with web/ to avoid confusion.

Yarn

This web app is managed with the package manager Yarn. This package manager uses the same package.json.

:::{tip} Some module not found errors can be resolved by deleting node_modules and running yarn again. :::

Yarn Scripts

  • dev: Runs the Vite development server on the local network.
  • build: Builds the Web UI using vite.
  • preview: Preview the built web app without injected node server.
  • start: Runs the built web app with the injected node server.
  • camera: Runs the camera server (variant with -online downloads required dependencies)