Lightingale Documentation

Welcome to the documentation hub of the Lightingale Community. See the sidebar for the list of documentations available.

To see a list of active projects not included here, feel free to visit the project website for the whole list, or GitHub pages host for projects hosted on GitHub.

Site powered by mdBook.

Clearnet CDNs are generously provided by Cloudflare, GCoreLabs, CloudFront, CacheFly and Fastly free of charge.

Services

For shorter descriptions, see Lightingale Services List.

• Lavender: Reddit-like Fediverse instance for the MLP fandom and LTGC.
• Liquestria: Invite-only semi-anarchy Minecraft survival multiplayer on mixnet.
• Matrix services: Managed Matrix bridges and more.

Projects

POSIX Shell

• Gel: ⛏ Rolling server bases, batteries included.
• Hyacinth: 💨 Build web projects with speed.
• Iceflakes: ❄️ Offline Linux installer images for everyone.
• Nightglow: 🌙 Yet another Cloudflare WARP client wrapper.
• shx: 📜 Bash/PDKSH shell action executor.
• Stratus: ☕️ JavaScript runtime containers (Deno, Bun, Node.js).

Data sheets

• midi-db: 🎹 Data concerning MIDI standards.
• Raven: 💨 Lightning-fast intuitive typing.
• SynPix: 👾 The open pixel font for embedded LCDs.

Rust

• Ensemble: 🎶 A synth framework in Rust.

Go

• Floaty: ☁️ Prevent Caddy loopbacks... Without traceability.

Java

• Mogician:
• Seperatist: 🔨 Seperated... As they should be.

Kotlin

JavaScript

• Berry: 🍇 Simple port exposure tunnel for stream entropy manipulation.
• Bread: 📦 Unified framework to stream binary in blocks.
• Crystal Quartz: 🎨 A colour palette generation system.
• Ditzy: 📬 Reconstruct stateful sockets over stateless messages.
• Eclipsed: 🔊 The Royal Canterlot Voice for SSE/EventSource.

• Ink Stone: 🖼 Modify font files with ease.
• Inkwell: 📒 The grand central terminal / Le grand terminal central.
• Leaflet: ⚠️ Stock Caddy error pages, free for anyone to use if kept credits.
• Lightfelt: 🕸 A collection of JS snippets for the Web and Deno.
• Mint: 🌱 Easy-to-configure load balancing serverless functions.
• Minuette: ⏳ Intercept, inspect, orchestrate.
• Octavia: 🎻 Event-driven multi-standard MIDI state-tracking library.
• Painted Palette: 🎨 Painting palettes, one pixel at a time.
• Parchment: 📃 Mark-down based client-side page generator.
• Rochelle: 🔪 Stream chunk splitting.
• Rosegap: 🌹🌙 Pleasant ads without compromises.
• Scope: 🔭 Easy-to-setup WireGuard meshing.
• Silk: 🕸 Bringing the fandom together.
• Snowy: ❄️ The BroadcastChannel polyfill for Firefox 29+ and Chrome 5+.
• Track me, senpai!: 🍑 Just giving thanks to your visitor sharing their everything with you.
• Twinkle Sprinkle: 📜 Task wait signals and simple WAL.
• WingBlade: ☁️ One codebase, multiple runtimes.

Privacy policy

As of 6th Oct 2023, these terms are not yet in effect.

By using services provided by the Lightingale Community, you agree to the following terms.

Default

Unless stated explicitly otherwise, the following policy will be applied to any web-based service run by the Lightingale Community.

TL;DR, we have access to all of the data the client implementation chooses to share with us, but unless an error is caused or you explicitly want us to, none of the data would be stored.

Accessible

Below is the list of data accessible by us if provided, on the server-side without user interaction.

• When accessing via clearnet, the IP address of your exit.
• When accessing via clearnet, IP packet fingerprints. (e.g. TCP fingerprint, UDP fingerprint)
• TLS connection fingerprint, when connected via TLS.
• Standardized client-identifying headers, if provided. Visit MDN for more information. Below is a list of possible headers, and may not include all headers.
  • Accept: Which types of content is accepted by the client.
  • Accept-Encoding: Encoding schemes accepted by the client, usually compression algorithms.
  • Accept-Language: Languages expected by the client.
  • Authorization: Tokens proving client's identification.
  • Origin: The URL of request's origin.
  • Referer: The initiator URL of each request.
  • Sec-Fetch-Dest: The initiator type of each request.
  • Sec-Fetch-Mode: Mode applied on requests.
  • Sec-Fetch-Site: Relationship between requested resource and source.
  • User-Agent: Implementation identification string provided by the client.
  • X-Forwarded-For, X-Real-IP: If behind a reverse proxy, the connecting IP address observed by the reverse proxy. These headers are almost always stripped, spoofed or generalized on reverse proxies we have control of however.
• Client hints, sent by Chromium-based browsers (e.g. Google Chrome). Below is a list of headers sent unconditionally by Chromium-based browsers. To prevent us from receiving these headers, please switch to a browser that isn't based on Chromium.
  • Sec-CH-UA: Array of implementation identification strings provided by the client.
  • Sec-CH-UA-Mobile: If the client runs on mobile platforms (e.g. Android, iOS).
  • Sec-CH-UA-Platform: The platform running the client, usually OS families.
• All other headers clients choose to provide.
• When the service involves server-side dynamically-generated content, basic cookies allowing said service to function properly.

We also have access to data you explicitly choose to share with us. They may be in cleartext or in their encrypted form.

Use

General

• Data explicitly stored by users will likely persist until takedowns.
• During suspected infiltrations, we will use any information accessible to identify and exterminate unauthorized access. Data stored during this period will be removed as soon as related investigations are finished.
• We do not share any information with any third-parties.
• Data requests from individuals and law enforcement agencies will be assessed before any action is taken. See the data request section for details.

Low-importance

Unless overriden, these uses do not store data.

• To deny access from clients we deem as unfit of service (WAF), we will use Origin, Referer, Sec-CH-UA, Sec-Fetch-Site, User-Agent, observed client IP address (if on clearnet/behind reverse proxies), IP packet fingerprints (if on clearnet) and TLS fingerprint.
• To conserve bandwidth whenever we can, we will use Accept-Encoding.
• To serve content to client's expectation, we will use Accept.
• To distinguish between sessions if essential to the service client accesses, Authorization and/or basic cookies will be used.
• To serve localized content in client's desired languages, we may use Accept-Language.

Medium-importance

Data stored from these uses will be kept for at most 7 days.

• Requests will be logged without observed IP addresses only upon causing non-critical software errors.

High-importance

Data stored from these uses will be kept for at most 365 days.

• Requests will be logged with observed IP addresses only upon causing critical software errors, or attempting to utilize known exploits.

Data access/removal request

• Approved data requests will be fulfilled within 30 days since approval, in the form that's available to us. Encrypted data will be provided as-is, along with their respective decryption keys only if available.
• Individuals can file requests for their data, and will be approved upon proving ownership.
  • In most cases, individuals have power to remove data on their own.
• Law enforcement agencies can file requests for data. All data requests filed from law enforcement agencies will be made public.
  • Requests sent from law enforcement agencies will only be approved if all listed criteria are satisfied.
    • Belongs to a political body deemed as "full democracy" or "flawed democracy" by The Economist Democracy Index.
    • Provides sufficient information for proving their identity.
    • Belongs to the political body where related services are hosted, or can provide proof that affected users violated our acceptable usage policy.
  • Affected resources of fulfilled removal requests from law enforcement agencies will be marked as 451 Resource Aflame, along with the name of requested agency.
• If the service's ability to continue operation becomes next-to-impossible due to a failed compliance from a rejected data request, we will wipe all data stored on our behalf and cease operation on said service. The same goes for the Lightingale Community itself.

Infrastructure providers

We extensively rely on Cloudflare for our infrastructure on the clearnet. If you do not want to go through Cloudflare, feel free to access our services through alternative means. Otherwise, a copy of privacy policy from Cloudflare could be obtained.

To improve availability and/or decrease cost on some of our services, we may also use other infrastructure providers. Check their respective privacy policies for details. Below is a list of providers we rely on.

• AWS CloudFront, for CDN.
  • Anonymized/generalized data generated by AWS CloudFront are stored for 30 days by the provider.
• AWS Lambda, for serverless computing.
  • Anonymized/generalized data generated by AWS Lambda are stored for 30 days by the provider.
• CacheFly, for CDN.
  • Data generated by CacheFly are never stored.
• Cloudflare, for DNS, CDN, WAF and edge serverless computing.
  • Identifying data generated by Cloudflare are stored for 24 hours by the provider.
  • Anonymized/generalized data generated by Cloudflare are stored for 30 days by the provider.
• Deno Deploy, for edge serverless computing.
  • Anonymized/generalized data generated by Deno Deploy are stored for 30 days by the provider.
• Fastly, for CDN and edge serverless computing.
  • Data generated by Fastly are never stored.
• GCoreLabs, for CDN.
  • Anonymized/generalized data generated by GCoreLabs are stored for 30 days by the provider.

Access denial

VPN

If you're accessing any Lightingale service from a VPN, chances are you're already blocked by us, but will get unblocked if with mixnets like Tor, or encrypted proxies. This is intentional.

We strongly support privacy and anonimity, while also highly detest the idea of fattening any commercial VPN vendor. VPNs do not offer privacy or security on the Internet in any way, let alone either anonimity or anti-censorship, and in no way any of them contribute to the effort of developing either anti-censorship or concealing technology to truly liberate the masses. As such, we've long since decided to take direct action, actively identifying and fingerprinting VPNs, while advising people to switch to real solutions instead.

Client

We identify some client programs as high-risk, and block requests from those programs if identified.

FAQ

I'm using a VPN, what should I do?

To put it simply, please stop using a VPN to access us.

If you're familiar with accessing services hosted on mixnets, you can access our services via alternative means. Or if you don't want to go through the hassle, please disconnect from your VPN, and if you have access to either exit/outproxy-enabled mixnet (e.g. Tor, I2P (I2Pd is recommended if you are comfortable with command line), Lokinet) or encrypted proxies, use them instead.

What if I'm blocked wrongly?

Within 18 hours, please contact us and provide your Cloudflare Ray ID in your unblocking appeal. After we've verified that you did indeed not connect from a VPN, you'll be unblocked as soon as possible. However, if you used a client program we deem as high-risk, we will reject your appeal immediately.

Could you stop blocking VPNs?

No.

Acceptable Usage Policy

These terms do not yet have legal liabilities, however they are enforced.

Underlined terms should have definitions available in the Definitions section.

Definitions

• Child pornography: Depictions of underage humans or underage characters resembling humans.
• Underage: Below 18 years old.

Disallowed use

AUP does not cover cases where reasons of policy enforcements are explained.

• Reselling any of our public services, which is free-of-charge for all.
• Breaching the awareness scope defined by projects/services.
• Scanning us or to scan others (e.g. port scanning).
• Gaining unauthorized resource access.
• Degrading the performance of our servers (e.g. flooding).
• Violating others privacy (e.g. doxxing).
• Impersonating existing identities without stating satire status.
• Exploiting stolen identities.
• Distributing unsolicited data (e.g. spam).
• Scamming.
• Hate speech (e.g. racism, xenophobia).
• Promotion of self-harm (e.g. suicide).
• Harming others physically or psychologicaly (e.g. torture threats, death threats, harassement).
• Engaging in distribution of illegal substances, human organs, guns and etc.
• Promoting, encouraging and/or beautifying any of the following topics.
  • Crimes against humanity (e.g. massacre).
  • Terrorism, its practices and exercising organizations.
• Distributing content matching any of the following criteria, without explicit warning to the target audience.
  • Malware.
  • Phishing content.
  • Pornography.
• Distributing and storing content matching any of the following criteria.
  • Child pornography.
  • Pornography without consent of involved parties.
• Defaming us without any validated proof.

Terms of Service

This agreement will become legally binding once Lightingale Community becomes a legally-registered non-profit organization.

As of 6th Oct 2023, these terms are not yet in effect.

In this agreement...

• The term "you" or "your" refer to "the user".
• The term "we", "us" or "our" refer to Lightingale Community itself and its members.
• The term "service" or "services" refer to publicly-available listed services hosted on behalf of Lightingale Community. The list is available here.

By accessing content or using the services provided by the Lightingale Community, you agree to the following terms. Specific services will apply amendments to fit their specific needs.

Any material changes to these terms will be publicly notified via our Mastodon and our Telegram channel at least 7 days prior. If you wish to exercise your right to reject such changes, you can file complaints to us, or stop using the service altogether.

Disclaimer

• We reserve the right to modify the service, including but not limited to changing certain features and updating the related software.

• We reserve the right to terminate your access to some or all of our services.

• We may deny access to certain services based on the geographic location of your immediate connection.

• We will attempt to prevent disruptions, defects and losses of data in our offered services, but despite our best efforts, we do not and can not guarantee availability of all times, and we are not responsible for materials served by us. Our service comes with absolutely no warranty, to the extent permitted by applicable law. You are solely responsible for your use of service.

The service

This section is defined in different services we offer to the public.

Software

Lightingale Community is powered fully by and produces FOSS, fit under the definition set by the Free Software Foundation. Further acknowledgements may be made in README files when we use software projects that require them.

Our content

Except for software, unless stated otherwise explicitly, content belongs to us is licensed under CC BY-NC-SA 4.0 License. Our members hold the rights of content they produced.

Your content

This section applies to services offering UGC.

You hold the rights to your own content, but you give us a license when submitting your content to us. Upon submission, you permit us to do the following to your submitted content. These terms are world-wide, non-exclusive, royalty-free, transferrable and perpetual.

• To store and display your content, you permit us to use, copy, store, distribute your content in ways consistent with your use of the services.
• To adapt your content for use of different platforms by different users with various backgrounds, you permit us to monitor, modify, translate and reformat your content.
• For public availability, you permit us to publish, publicly perform, or publicly display your content, if you've chosen to make it visible to others.
• To allow our services to work as intended, you permit us to sublicense your content.

We reserve the right to block, remove or wipe your content if in breach with these terms or other terms in effect.

Acceptable usage

As a user of services offered by us, you agree to adhere to our acceptable usage policy. Violations of AUP will result in a termination, and may result in your data made available to law enforcement agencies.

We encourage you to report abuses of our services with proof. We accept reports made via email at abuse (at) ltgc (dot) cc.

Donation

This section is subject to changes as our legal status changes.

We are not responsible for fulfilling any additional requirement attached to the donations made to us.

Applicable law

This section is subject to change as our legal status changes.

Legal contact

This section is subject to change as our legal status changes.

Legal contacts may be filed via email to legal (at) ltgc (dot) cc. We do not have a physical location available.

UGC Availability

To prevent Lightingale Community from getting blocked as a whole in certain countries, we've been forced to restrict or deny direct access from those countries. A restricted service coverage would've been marginally better than nothing at all.

We offer two levels of UGC services: high-liberty and restricted. Levels are separated by the measures avalable to the users, and users' ability to freely express whatever they desire. Some countries may have been denied direct access from high-liberty UGC services, but not restricted ones.

If you've been redirected to this page, it's likely that you have been denied direct access to our UGC service, and we have no other choice but to offer our deepest apology.

List of countries affected

Denied from high-liberty and restricted UGC services

• AE: United Arab Emirates
• AF: Afghanistan
• AZ: Azerbaijan
• BD: Bangladesh
• BH: Bahrain
• BT: Bhutan
• BY: Belarus
• CN: China
• CU: Cuba
• DJ: Djibouti
• DZ: Algeria
• EG: Egypt
• ER: Eritrea
• ET: Ethiopia
• GT: Guatemala
• HN: Honduras
• KH: Cambodia
• KZ: Kazakhstan
• IL: Israel
• IN: India
• IQ: Iraq
• IR: Iran
• LA: Laos
• LB: Lebanon
• LY: Libya
• MM: Myanmar
• NI: Nicaragua
• NK (KP): North Korea
• OM: Oman
• PK: Pakistan
• PS: Palestine
• RU: Russia
• RW: Rwanda
• SA: Saudi Arabia
• SD: Sudan
• SL: Sri Lanka
• SO: Somalia
• SY: Syria
• TJ: Tajikstan
• TM: Turkmenistan
• TR: Turkey
• UZ: Uzbekistan
• VE: Venezuela
• VN: Vietnam
• YE: Yemen

Denied from high-liberty UGC services

• BO: Bolivia
• CD: Democratic Republic of Congo
• CM: Cameroon
• GQ: Equatorial Guinea
• HK: Hong Kong
• KG: Kyrgyzstan
• JO: Jordan
• KW: Kuwait
• LS: Lesotho
• MA: Morocco
• MX: Mexico
• PE: Peru
• PH: Philippines
• SG: Singapore
• SS: South Sudan
• SV: El Salvador
• UG: Uganda

Technologies

Preferred technologies

Languages and runtimes

• Bash
• Deno
• Go
• Rust

Networking

• CoreDNS
• I2Pd
• IPFS
• Lokinet
• Tor
• WireGuard
• Yggdrasil

Operating systems

• AlmaLinux
• Alpine Linux
• Debian
• openSUSE
• Rocky Linux

Virtualization and containerization

• KVM
• LXC
• Podman

Miscellaneous

• Caddy
• FabricMC
• Nixpkgs
• Syncthing

Used technologies

Technologies listed here may have been used by some of our projects. None of the technologies listed below are a preference, though project can freely choose to use them.

Languages and runtimes

• C (isolated wherever possible)
  • Clang
  • GCC
• C#
• Dart
• Java
  • OpenJDK
  • Microsoft OpenJDK
• Kotlin
• WebAssembly
  • wasmtime
• Zig

Virtualization and containerization

• Docker
• Xen

Miscellaneous

• PostgreSQL

Avoided technologies

Technologies listed here will have their usage be discouraged and avoided. However, projects can still develop for them only if necessary. Software already developed with these technologies can also be deployed.

Languages and runtimes

• Node.js
• PHP
• Python
• TypeScript

Operating systems

• Ubuntu

Miscellaneous

• Snap

Banned technologies

Usage of technologies listed here are entirely banned within LTGC. Bans do not extend to members themselves.

Frameworks

• ASP.NET
• JSP

Virtualization and containerization

• OpenVZ

Miscellaneous

• LXD

Content encoding

None of the encodings listed are involved with patent concerns.

Read below for further details.

Rasterized image encoding

Lossy still

As WebP is properly supported in every browser built for systems newer than Windows XP, and it offers better quality-to-size ratio than all JPEG encoders, it is considered the best fallback format. However, as JPEG XL exists as an all-around better universal image format, lossy images should also be encoded with JPEG XL whenever possible.

Compared to all other image formats, JPEG XL offers jawdropping image fidelity, unmatched quality-to-size ratio, proper progressive loading and better responsiveness. The cherry on top of JPEG XL would be its ability to losslessly transcode from existing JPEG files, offering about 20% size reduction with cheap JPEG lossless bitstream reconstruction, which paves its way to be a backwards-compatible drop-in replacement for image storage and distribution.

Things to take notice of:

• mozJPEG-fronted JPEG, while offering a good choice for backwards-compatibility, shouldn't be used unless necessary when size is a consideration. mozJPEG is generally superseded by jpegli.
• AVIF offers an advantage on size, when progressive support, boundary artifacts and codec performance (slower than all others multiple times) aren't in consideration.
• AVIF is observed to have fewer observable artifacts for images with a simpler shading.
• The reference AVIF settings provide a balancing point between JXL -d 1 and JXL -d 2.
• jpegli for JPEG from the JPEG XL team provides a reasonable front for backwards-compatible JPEG XL encoding. The estimated file size on fronted JPEG XL files are around 82.5% of the original JPEG files.

Explanations of image encoding uses:

• Delivery: Size-first with generally-acceptable quality.
• Delivery+: Balance between size and quality.
• Near-lossless: Shaving as much size as possible, while still being mostly indistinguishable, even when zoomed in at 200%.

The size ratio values listed below approach expectations on a large scale, based on experiments conducted on a relatively large scale with non-photographic images modelling 15.6% lossy inputs. Lossless inputs induce better (smaller) expectations, while lossy inputs tip the balance in reverse. Photograpic images will generally have a better (smaller) ratio than non-photographic images, but individual images may still yield greatly varied results.

The full feature set comparison chart is available on Cloudinary.

Lossless still

JPEG XL lossless and WebP lossless are mostly on par, either could be the better solution at times. For compatibility reasons, WebP lossless should always be offered, while JPEG XL lossless could be offered only when the resulting file is smaller than that of WebP lossless. AVIF lossless is currently consistently worse than either JPEG XL or WebP, as such it's not recommended at all.

Do not enable progressive encoding for JPEG XL lossless, or the resulting file will be several times bigger than PNG files.

Lossy animated

Lossy animated WebP is the current baseline for animated image sequences, while lossy animated AVIF offers the best quality against others by a wide margin. Lossy animated JPEG XL doesn't offer a significant advantage against WebP, and is currently beaten by AVIF.

Audio encoding

Lossy

When high sampling rates are required, choose Vorbis. When support for the Apple ecosystem is required, choose AAC-LC. Otherwise use Opus under all possible scenarios, but beware that Opus only supports sampling at 48kHz.

Below are the suggested bitrates under different scenarios, when encoding stereo audio content under either 44.1kHz or 48kHz. Audio content should be encoded with constrained variable bitrate (CVBR).

The AAC-LC encoder in question is libfdk_aac, being the best FOSS AAC-LC encoder out there. The only AAC-LC encoder better than libfdk_aac is Apple's Audio Toolbox, which is proprietary and not in consideration.

Keep in mind that the scenario under "basic" indicates that, all audio content encoded with provided parameters should be virtually indistinguishable from LAME-encoded MP3 files at 192kbps.

Lossless

Streaming lossless audio is generally considered a bad idea due to high bandwidth usage. It's generally more useful for archival purposes.

You should choose FLAC when...

• Browser support is needed.
• The target is severely underpowered.
• The target only supports FLAC for lossless audio compression.

You should choose WavPack when...

• You need a smaller file.
• Higher bit depth is needed (more than 24).

Additional considerations:

• If you do not intend to distribute high bit-depth lossless audio, but the input you have is of high bit-depth, you can re-sample it to somewhere between 16-bit (-96 dB) and 24-bit (-144 dB). Use --pre-quantize=16 for WavPack.

Compression

Web

Brotli should be preferred over gzip at all times. Static low-entropy content (e.g. plain text files) should always be pre-compressed, with or without the original uncompressed file available. For dynamic content compression, zstd at level 8 consumes significantly less than the resources used for Brotli, all the while outperforming Brotli in terms of output size.

For static precompression, the original file can be omitted when the space is constrained, serving only precompressed blobs. When compatibility with other infrastructure isn't in consideration, precompressed gzip files can also be omitted, although it's not recommended in most cases. zstd isn't suitable for static precompression, as even at compression level 22 it falls behind Brotli at its highest compression level. However, a recommendation is still available for reasonable reference. Zopfli is an alternative compressor for gzip, which --i1 outperforms gzip even at level 9, however only --i1 and --i2 are recommeded for general use.

The table below provides a quick reference of levels and estimated size reduction by use case.

On a case-by-case basis, Compression Tester could be used to see algorithms in action. For a more comprehensive comparison, Morotti's Compression Results could be used.

Bundle

lzip is the de-facto standard of bundle compression, featuring good compression ratio with data recovery abilities. Quality should always set to 9, unless a lower quality value is proven to yield a better result for smaller files, or time constraints are set.

bzip2 or xz should only be considered for compatibility purposes.

Font

As practically all modern browsers support the WOFF2 font format, all required TrueType and OpenType fonts must be served as WOFF2 files instead. If for unspecified reasons that original files must be served, always consider precompression and avoid storing uncompressed files.

Region and DC codes

Datacentre codes

Versioning

Semantic versioning

Visit semver.org for more information.

Decaday versioning

Inspired by Minecraft's snapshot versioning appraoch, decaday versioning is used for projects releasing snapshots, where the versioning follows the YYYYdDDx, where YYYY denotes the full year, d can either be d or m in different projects (due to historical mistakes...), DD denotes the decaday of the year, and x denotes the ever-increasing alphabetical index of snapshot versions.

"Decaday" is a way of dividing months to three roughly equal portions, where it usually tries to group 10 days as a "decaday". In a typical month, the 1st to the 10th day belongs to the first decaday, the 11th to the 20th belongs to the second, and the 21st to the 30th belongs to the third. If a month has 31 days, the 31st day belongs to the third decaday. This approach avoids the problem of different choices in a week's starting point.

Take 2023d30b as an example. 2023 means the snapshot was released in 2023, 30 means the snapshot was released between 21st October and 31st October, and b indicates the version was released as the second snapshot between the time period.

Project origins

What led to LTGC projects?

• Thestral: "Is oblivious user authentication possible?"
• Nightglow: "Ugh the official Cloudflare WARP client is so crappy and I don't need that much functionality anyways!"
• Scope: "No existing meshed WireGuard orchestration fits our needs!"
• Octavia: "Can I make a better MIDI visualizer?"
• shx: "I hate Node.js but I need something like npx."
• SynPix: "Emulated screens bundled in Octavia need a libre bitmap font."
• MIDI DB: "Why isn't there readily-available multi-standard MIDI data sheet?"
• Hyacinth: "JS projects should be built faster!"
• Painted Palette: "What if I made a better headless bot?"
• WingBlade: "Can I make the same codebase run everywhere?"
• (unnamed): "Let's build a clean, lightweight and ad-free boardgame experience!"
• Linen: "Wanna run an r/place replica of our own?"
• Cubics: "Want a convenient diagram generator for Rubik's cube-like puzzles..."
• Raven: "I don't want to relearn a keyboard layout to type in another writing system."
• (unnamed): "Wanna optimize Minecraft?"
• Rosegap: "Daily Dose of Pony needs an ad system."
• Ink Stone: "Will SynPix ever be released as font files?"

Where did names of projects come from?

Project inspiration hiearchy

• Project Moonlight (Media)
  • Heartstrings
  • Octavia
    • Cambiare
    • Ensemble
    • Hyacinth
      • shx
    • MIDI DB
    • Snowy
    • Swiftbook
    • SynPix
      • Ink Stone
• Project Lilac (Translation, localization and linguistics)
  • Noitieng
  • Raven
• Project Weaving (Networking and isolation)
  • Floaty
  • Nightglow
  • Patchwork
  • Scope
• Uncategorized
  • Acari
  • Berry
    • Bread
  • Citrus
  • Crystal Quartz
  • Daily Scoop
  • Ditzy
  • Domain DB
  • Flex
  • Flitter
  • Graceful Lattice
  • Inkwell
  • Leaflet
  • LiteBlock
  • Mint
    • Flame
  • Minuette
  • Painted Palette
    • WingBlade
      • Eclipsed
      • Rochelle
      • Stratus
      • Twinkle Sprinkles
  • Patchwork
  • PonyTrails
    • Gel
      • Iceflakes
  • Rosegap
  • Seperatist
  • Silk
  • Thestral

List of technologies

JavaScript

Engines

Runtimes

List of sizes of data types

Numbers

Integers

• Aliases for certain integer types:
  • u8: byte (Go)
  • i32: rune (Go)
  • bool: _Bool (C)
• GNU libc offers alternative types for intergers. Read more

Floats and more

• The C# decimal type occupies 128 bits, but is different from f128.
• long double in C varies in compiler implementations.
• Some C implementations may use _Float128 for f128.

Lavender

Lavender is a friendly discussion lounge for both the My Little Pony fandom and LTGC, powered by the magic of Lemmy. No matter what you are, be a pony, dragon, hippogriff or draconnequus, you're welcome here!

Lavender adheres to the general Terms of Service and Acceptable Usage Policy.

The incubator thread can be viewed here.

Access

Addresses

The service can be accessed from the addresses listed below.

• Clearnet (not recommended): lavender.ltgc.cc
• Tor: lavender...j33btaqd.onion
• I2P (I2Pd): lavender.ltgc.i2p or lavender...v6bg4rfa.b32.i2p
• Lokinet: mwt5jdqg...jpxctyeo.loki
• Yggdrasil: lav.ygg.ltgc.cc

Panels

Read Lavender panels.

Account registration

Alpha phase

Lavender is still in the alpha phase. During the alpha phase...

• Only people already having a presence on Fediverse can apply for account registration. You must fulfill any criteria listed below.
  • Have an account on one of the listed public instances for more than 14 days, with at least 7 posts in total.
  • Have an account on any public Fediverse instances for more than 14 days, with at least 7 posts in total, and the identity relates to the My Little Pony fandom.
  • Be a member of the Lightingale Community.

Moderation

Alpha phase

Lavender is still in the alpha phase. During the alpha phase...

• Only accounts on the instance itself can apply to become a moderator.
• Only with two or more

Lavender Panels

Lavender currently organizes the following panels.

Best of Fedi Pones

Artists new to the Fediverse often complain about not reaching enough audiences. So what's the solution? Well, with a weekly panel promoting pony artworks published on the Fediverse, of course!

Organized by a group of Fediverse enthusiasts from the My Little Pony fandom, Best of Fedi Pones seeks to bring what the Fediverse has to offer to everyone enjoying the creativity from the pony fandom. Better together!

Source code: Codeberg

Artwork submission

Submitting artworks will get them included in the issues of Best of Fedi Pones if not denied. All submitted artworks, unless violating AUP, will be also referenced by the MLP English Lavender community.

Submitting artworks

Best of Fedi Pones has been directly integrated into the Fediverse since issue 10. Submission only requires the inclusion of #weeklypony tag while also mentioning @[email protected]. Works both when posting yourself and replying to others.

Inclusion denial

Submissions will be denied if they match at least one of criteria listed below.

• Is not SFW.
• Is generated by neural networks. Inspired by or assisted with NN doesn't count.
• Not posted by the artist themself.
• Violates LTGC AUP.
• Was posted for longer than 14 days to the deadline of the target issue.

Voting and issuing

Voting of each submission is done in a room accessible to all panel organizers. While the room is bridged to Discord and Telegram, only votes from Matrix and Discord are counted due to technical limitations.

Each eligible voter can part one ⭐️(:star:) emoji to give a submission an upvote. Submissions with high vote count will be ordered first, and submissions sharing the same vote count have their exact order determined by the hash of the origin URL. No downvotes are available.

Voting of each issue generally begins at 12:00 UTC +0 on every Saturday, and ends at 18:00 UTC +0 on the next day. The dataset should be available shortly after the voting ends, and any organizer can release the issue with the dataset.

To prevent accusations of theft, except for the cover art, all submission images must be hotlinked from source. Cover art is generally selected among the submission with the top 60% vote count, entirely by the organizer posting the issue with their own preferences. However, it's advised to avoid featuring the same artist within the span of 9 issues, and artists new to the Fediverse are advised to be preferred.

IM services

Lightingale Community has been hosting messaging services like Matrix for ease of communication, ever since the community's early days in 2021.

Services

Matrix

We run a Synapse homeserver for hosting all of our bridges and for internal communication. Only members of the Lightingale Community can apply for accounts on the homeserver.

Mumble

Planned. Internal and mixnet access only, with no clearnet access planned.

XMPP

We run a Prosody server with native mixnet access, for easier interfacing with the federated XMPP ecosystem. It is not intended for primary use, and its service availability is generally worse than Matrix.

Bridges

As long as adhereing to the Acceptable Usage Policy, members on our homeserver can freely use any of the bridges we host. Certain bridges offer integration to mixnets and overlay networks, namely I2P, Tor and Yggdrasil.

However, please note that integrations with alternative DNS systems (e.g. OpenNIC, ALFIS) likely will not work.

Group chat bridges

The following bridges support both group chats and direct messages.

• Discord to Matrix (via mautrix-discord)
• IRC to Matrix (via Heisenbridge) (planned)
• Mumble to Matrix (via MandM-bridge) (planned)
• Telegram to Matrix (via mautrix-telegram)
• XMPP/Jabber to Matrix (via matrix-bifrost) (planned)

Direct message bridges

The following bridges only support direct messages.

• Facebook Messenger to Matrix (via mautrix-meta) (planned)
• Instagram to Matrix (via mautrix-meta) (planned)
• Twitter to Matrix (via mautrix-twitter)

Voice chat bridges

The following bridges only bridge voice chats.

• Mumble and Discord (via mumble-discord-bridge) (planned)

Media proxies

Active Discord media proxy

As Discord began requiring expiring HMAC validations on media attachments, the Discord bridge now switches to an active proxying approach, before reuploading/re-serving re-encoded optimized media attachments becomes viable. Files delivered via this new approach will be served at dma.ltgc.cc.

However due to poor design, the active Discord media proxy fails from time to time. We are working on a dedicated active media proxy written in either JS (Deno) or Go.

Passive Discord media proxy

Due to MSC3860 being unsupported on homeservers without conformation of the Matrix 1.7 specification, homeservers will not recognize discord-media.mau.dev, the public redirecting Discord media repo. To circumvent this, we offer a custom cached Discord media proxy at dmr.ltgc.cc instead for use with bridges, however it will likely be shut down to public access if we notice it getting abused.

After Discord implemented expirations on media attachment links with HMAC validation, the media proxy now only serves non-attachment files, like avatars, stickers and custom emojis. Since passive media proxy is less costly to run than active media proxies, it's used whenever possible.

Bridge utilisation

Apply for use

Apart from internal use, we also offer bridging services to other entities upon approval with no SLA.

FLOSS projects

• RethinkDNS (Apache 2.0, MPL 2.0)
• Xray (MPL)

MLP fandom communities

Not-for-profit entities

• Uncyclopedia (Traditional Chinese)

Known issues

Global

• Animated stickers on certain platforms may only get their first frame be bridged, due to varying media codec support on different platforms.
• Double-bridged replies may not work on certain platforms.
• The bridge may become unresponsive when overloaded. You can help us relieve this problem via donations!

Telegram

• Reactions aren't bridged from Telegram.
• The bridge suffers from unresponsiveness or even downtimes due to PostgreSQL. Migration to SQLite3 is currently scheduled.
• Bot messages from Telegram cannot be bridged to other platforms, per Telegram's restriction.
• Senders from other platforms cannot be transparently bridged to Telegram, requiring embedding their display names as message content.
• Member leave events aren't bridged to Matrix, as such the Telegram member list on Matrix is not representative.

Discord

• Due to Discord validating time-sensitive HMAC codes on attachment URLs, media from Discord cannot be bridged currently.
  • This was fixed on 27th March 2024, where a combination of active and passive media proxying was deployed.
• Some of the animated stickers were vector images encoded in custom-defined JSON files. As a result, none of the services other than Discord recognizes them.
• The Discord bridge invites bridged accounts to rooms on-demand, and does not bridge member leave events to Matrix, as such the Discord member list on Matrix is not representative.

Ditzy

Ditzy is a transport-agnostic universal multi-message encoding scheme, allowing stateful bidirectional communications regardless of the statefulness of the underlying transport. With the state of the reconstructed tunnel fully decoupled from the underlying transport, possibilities are endless. How Ditzy performs is entirely dependent on implementation.

This project is inspired by Tor HTTP Meek, MIDI 1.0, Reliable UDP and QUIC.

💌 Feel your messages delivered safely by your trustworthy Ponyvillan mailmare!

The first draft was written on 18th Nov 2021, with the next iteration on 5th Feb 2023. The current draft is written on 29th June 2023.

Advantages

• State of reconstructed connections are decoupled from their underlying transports entirely
• Seamless connection migration on connection losses and IP changes
• Total control over data transmission

Disadvantages

• Directional speed penalty when not in stream mode
• Not suitable for time-sensitive applications
• Resource overhead on both servers and clients
• Possible attack vector via decorated messages

Use cases

• Bidirectional communication through incapable infrastructure (e.g. CDNs, serverless functions, request reflectors, fronted containers)
• Connection multiplexing and splitting
• Stablizing communication over unstable networks
• Replacing QUIC where UDP communication is not feasible

Specifications

• Binary encoding format
• Recommended implementation practices

Reference implementation API docs

• JavaScript

Binary encoding format

Concepts

Meek

Originally, Meek is a technique emulating stateful connections over HTTP, utilized by Tor for its pluggable transports. It is now generalized to refer to any technique reconstructing stateful connections, regardless of the statefulness of underlying transport.

MIDI

MIDI is a standard for real-time communication between devices. It's originally designed for musical instruments, but has found its places in other fields requiring synchronization and/or automation as well, such as stage lighting systems. It could be thought of as a protocol of communicating changes about the states of instruments via a set of standardized messages.

Variable Length Value

Variable Length Value (VLV) is a way of encoding integer values to multiple bytes, utilizing as many bits inside as they want. With VLV, integer values can be as large as they want within reason, and reduce the byte count when the value is smaller with little overhead.

VLV is defined as a part of Standard MIDI File (SMF) Format Specification.

A VLV can take multiple bytes, and all data encoded are stored in Big-Endian format. VLV bytes use the bit before actual values of each byte to indicate whether or not to continue decoding, and the rest to carry actual values.

Examples below.

RUDP

RUDP is a general-purpose transport running on top of UDP, designed to achieve guaranteed and ordered packet delivery without the overhead TCP brings. The exact protocol is never documented, but RUDP will always have the following features:

• Acknowledgement of received packets
• Flow control and windowing
• Retransmission of lost packets
• Over-buffering

Ditzy attempts to implement some of the features, optionally with twists:

• Acknowledgement of last received packet IDs in each socket
• Retransmission when messages are considered lost
• Backpressure with customizable buffer/high watermark whenever possible
• Transmit-ahead window whenever possible

QUIC

QUIC is a general-purpose transport running on top of UDP, seeking to be a replacement over traditional TCP. It offers various advantages over traditional TCP, with the most notable ones being seamless connection migration and reduced communication latency.

Ditzy aims to achieve seamless connection migration via its client ID binding mechanism with minimal connection overhead.

Specification

Structure

Transmission

Client data socket
|
| Ditzy encoder
| |
| | Client transport
| | |
| | | (the state of the outermost transport is unrelated
| | |  to the states of the reconstructed data sockets)
| | v
| | Server transport
| v
| Ditzy decoder
v
Server data socket

Message

Each Ditzy message is formed by directly combining multiple frames together. The structure of each frame is as follows.

Socket ID

The socket ID is randomly selected between 1 and 268435455 (0xfffffff, 2²⁸ - 1).

Socket ID 0 is reserved for special purposes and is forbidden from transmitting any payload. It is used to deliver signals that could affect all connections of the client.

Frame ID

The frame ID is iterated from 0 to 268435455 (0xfffffff, 2²⁸ - 1) on the side where connections are initiated. When 268435455 is reached, the message ID is expected to roll back to 0. Frame ID should only be validated and increased in certain commands, check "Step?" to see if they undergo such process.

Length, EoP + Checksum

When Ditzy is set to fast mode (should only be used in safe environments), the payload length value only points to where the EoP byte is located. The EoP byte in fast mode only serves as a confirmation for bounds checking, and thus both sides should only select a random byte that has the 7th bit set to 0 (0~127) to do the job. When the supposed length doesn't match, Ditzy will discard all messages in the bundle completely in case of a decoration attack.

When Ditzy is set to strict mode, the payload length value is ignored, and Ditzy will attempt to validate the checksum once it finds the EoP byte. If the checksum does not match, Ditzy will discard all messages in the bundle completely in case of a decoration attack.

The checksum is calculated by setting the initial value to 63, and performing XOR operations to every byte in the payload. Once this is finished, the calculated value is then inverted, have 1 added. The final value would the the last 7 bits (0b0xxxxxxx) of the value. Example JavaScript code below.

let ditzyChecksum = (bytes) => {
	let checksum = 63;
	for (let i = 0; i < bytes.length; i ++) {
		checksum ^= bytes[i];
	};
	return (~checksum + 1) & 127;
};

Commands

IDs between 0 (0x00) and 31 (0x1f) are reserved for core functionalities. Extenstions are free to utilize any ID between 32 (0x20) and 255 (0xff).

Different commands have different support levels. To meet a certain support level, features of the indicated level must be implemented.

0: Socket close

Can be initiated bidirectionally.

Response should be exactly the same as request.

Closes a single connection. Payload can be used to carry optional arbitrary printable error messages, UTF-8 encoded. Analogous to 8x nn vv or 9x nn 00.

1: Socket open

Can be initiated bidirectionally.

Response should have the same frame ID as request.

Opens a connection. Payload is used to define suggested Socket timeout in milliseconds optionally. Frame ID must be set to 0, or the message will be considered invalid.

(Optional) If sent from the receiving end, the value contains would be the final timeout value used in the connection. Analogous to 9x nn vv.

2: Socket aftertouch/signalling

Can be initiated bidirectionally.

Response should have the same frame ID as request.

Challenges a connection. Often used to implement socket latency tests. Analogous to cx vv.

Payload sets challenge type.

Types of aftertouch

• 0: Latency test SYN1.
• 1: Latency test ACK1.
• 2: Latency test SYN2.
• 3: Latency test ACK2.

Types of signalling

• 128: Bind client ID, must be the very first message of every Ditzy connection. Frame ID contains the desired client ID. Each active connection can only have a client bound to an ID once, or the connection will be terminated to prevent flooding. Server response indicates a bound client ID.
• 129: Client terminate. Frame ID contains the desired client ID, should match the client ID bound to the transmission session. All sockets managed under the provided client ID will be gracefully closed. Server response indicates a successful queue of graceful client termination.
• 130: Saturated send buffer. Typically sent from the server to the client, this is used to signal the client to establish more connections with the server to help deplete its send buffer.

3: Jump

Can be initiated bidirectionally.

No responses are needed for jumps.

Sends a junk frame. The receiving end should ignore jump messages. Useful to implement noise.

4: Full payload send

Can be initiated bidirectionally.

Use type 5 (payload acknowledge) to send acknowledgements with the same frame ID.

Sends a full message. Analogous to f0 .. .. f7.

5: Frame acknowledge

Can be initiated bidirectionally. Only required over unreliable channels.

Sends acknowledgement of the last received sequencial frame.

6: Error

Can be initiated bidirectionally.

No response is needed for errors.

Sends arbitrary errors without closing the connection.

7: Implementation exclusive

Can be initiated bidirectionally.

Response should have the same frame ID as request.

Implementation exclusive binary commands. Analogous to f0 .. .. f7 or ff .. ..

8: Partial payload send

Can be initiated bidirectionally. Certain underlying transports may disable this.

Use type 5 (payload acknowledge) to send acknowledgements with the same frame ID.

Sends a segment of a message. Used to begin and continue sending the segments of a full message. The server should begin to interpret "full payload send" messages as "payload send complete" on the same socket. Analogous to f0 .. ...

9: Partial payload send continue

Can be initiated bidirectionally. Certain underlying transports may disable this.

Use type 5 (payload acknowledge) to send acknowledgements with the same frame ID.

Sends the final segment of a message. The server should interpret "full payload send" messages as "payload send complete" on the same socket. Analogous to f7 .. ...

10: Unordered tailing acknowledgement

Can be initiated bidirectionally. Only required over unreliable channels.

Sends a list of unordered received frames after the last continuously received frame. Used to reduce frame retransmissions when a copy has already been received.

After frame acknowledgement (message type 5) is sent, unordered tailing acknowledgements could be sent multiple times with the same frame ID as the preceding frame acknowledgement message, as long as the list of acknowledged frames aren't fully sent yet. Frame IDs are all encoded in 7-bit VLVs, with each VLV considered as an element of the list.

Recommended implementation practices

Ideas

Meek as a general terms, covers techniques reconstructing states of duplex sockets over underlying measures of communication. It doesn't have to be constrained to a single set of views. Under ideal circumstances, Meek can achieve full socket state decoupling with arbitrary socket delivery, however reaching those

Generic

Floaty

Floaty is a Caddy plugin implementing customizable rolling random IDs on a per-host basis, providing easy access via placeholders in Caddy.

Uses include:

• Loopback prevention
• Multi-hop debugging
• Forced time-based rolling sticky hash wherever needed

All with little to no traceability, if the rolling duration is set correctly.

Usage

Caddyfile

In the global section, set Floaty to run before request header processing.

{
	order floaty before header
}

In any of the server blocks, use the floaty directive to prepare the respective rolling random IDs. The syntax is as follows.

floaty [length [rollDuration]] [{
	[fieldId [length [rollDuration]]]
}]

Whenever Floaty is initialized, the placeholder http.floaty would become available within that server block, where placeholders from HTTP handlers are accessible.

Generated ID lengths can be any value between 4 and 96, and out-of-bound values will be clamped into this range. By default, length is set to 8. Longer IDs may cause excessive resource usage.

Roll duration can be set to any value above 10 seconds with millisecond precision, if supported by the Go duration syntax. It's set to 15 minutes by default. A lower rolling duration may cause excessive resource usage.

When a request is received, Floaty will attempt to check if any of the needed request IDs have expired. Only when expired upon receiving a request, will Floaty begin to regenerate a new ID. This will add a little bit of overhead to each request, however it will prevent unnecessary ID rerolls when not in use.

Examples

Adding a response header with a default Floaty ID:

http://:8080 {
	...
	floaty
	header X-Floaty {http.floaty}
	...
}

Adding a request header with a Floaty ID to reverse proxies, while blocking requests with a matching Floaty ID:

http://:8080 {
	...
	floaty {
		instance_id 12 10m
		source_id 12 5m
	}
	...
	@directLoopback {
		header X-Previous-Hop {http.floaty.instance_id}
	}
	@indirectLoopback {
		header X-Source-Hop {http.floaty.source_id}
	}
	respond @directLoopback "Loopback denied" 403
	respond @indirectLoopback "Loopback denied" 403
	...
	reverse_proxy {
		...
		header_up X-Previous-Hop {http.floaty.instance_id}
		header_up ?X-Source-Hop {http.floaty_source_id}
	}
	...
}

gel

⛏ Rolling server bases, batteries included.

Gel is a set of scripts to quickly prepare supported base Linux distributions for use in cloud, aiming to lessen the burden interacting with servers. For Alpine Linux, it comes with a simple OpenRC wrapper to get several systemctl commands working for OpenRC.

Gel is extensively used throughout Lightingale Community's infrastructure, either for host servers or for LXC containers. Read more on infrastructure.

Flavours

Gel mostly comes with two flavours for each supported base distribution: slim and full.

The full flavour targets all sorts of servers. Unless storage is severely constrained, this is the flavour to go for.

The slim flavour targets container hosts and VM hosts, which barely gets interacted directly, but still requires the ease-of-use found in the full version of Gel. It can also be used in other scenarios, if the full flavour contains packages you'd prefer not to be present on your system.

Full list

Usage & Security

• Installation and primitive security
• LXC setup

Installation

Native

For bare-metal, virtual machines and LXC containers. Do not run native install scripts other than on new machines.

1. On any of the supported distros, make sure curl is available.
2. Execute sh <(curl -Ls https://github.com/ltgcgo/gel/releases/latest/download/install.sh). (or sh <(curl -Ls https://codeberg.org/ltgc/gel/releases/download/latest/install.sh), use GEL_SLIM=1 to enable slim installation)
3. Connect via SSH with ssh -p 1122 <serverIP>. User passwords won't change, but SSH settings will. See the SSH section for details.

Containers

Container images are only offered as a convenient way of evaluating the installations.

1. Spin up one of the available Gel flavours.
   1. Images are available on the Docker Hub if you want to save time. Use podman pull docker.io/ltgc/gel:<flavour> to pull the images.
   2. Or feel free to build the image yourself with ./shx up <flavour>.
2. Connect to the SSH with ssh -p 1122 [email protected]. The default password is root.

Additional considerations

Offloading RAM

Warning

Avoid offloading RAM in guests as much as possible, when the root host you have access to of the guest could have RAM offloading enabled instead.

Gel does not configure RAM offloading automatically in any way.

swap

When enabled, the system will swap memory pages used less frequently off the active RAM, freeing up system resources whenever possible. This will lessen the impact of memory hungry processes, at the cost of disk wear and tear.

swap comes in two types, disk partition and files. Swapfiles are generally not supported in certain file systems like btrfs, however they're much more flexible compared to disk partitions, since they could be created without threatening the underlying filesystem and can have multiple active simultaneously, as such swapfiles should be preferred whenever possible.

Depending on the actual size of RAM and remaining vacancy on the disk, the chosen size of the swap is generally between 50% and 200% of it. Generally for a system with small RAM (e.g. 1 GB), a swap with the size of 2 GiB is desired; for a system with large RAM (e.g. 32 GB), a swap with the size of 16 GiB could be considered sufficient. If you want to have hibernation enabled on your system, the swap size should be no smaller than the physical RAM.

To create a 2 GiB swapfile at /swapfile (1048576 * 1024 * 2), run the following commands as root.

dd if=/dev/zero of=/swapfile bs=1048576 count=2048
mkswap /swapfile
chmod 600 /swapfile

To enable the swapfile immediately, run the following command. Disabling the swapfile only needs to change the swapon command to swapoff.

swapon /swapfile

If the swapfile at /swapfile needs to be loaded automatically across reboots, open /etc/fstab and append the following line.

/swapfile   none    swap    sw  0   0

zswap

Notice

All steps demonstrated below do not persist across reboots. Read Setting up Zswap in Debian 11 GNU/Linux for persistence.

Requires swap to be enabled. zswap creates a compressed cache in RAM for system to swap pages off to, before the system decides to offload to the on-disk swap instead. Enabling zswap on top of swap helps to lessen the burden dumped onto the disk, make memory usage more efficient, while increasing responsiveness, as some of the infrequently accessed pages now resides in the much-faster RAM instead.

To see the current configurations for zswap, run the following command.

grep -R . /sys/module/zswap/parameters

Stastistics for zswap is also available.

grep -R . /sys/kernel/debug/zswap

Enabling zswap is as easy as running the following command as root.

echo 1 > /sys/module/zswap/parameters/enabled

zswap will use at most 20% of your actual RAM to store the compressed cache. Let's say you want to adjust the allocation percentage to 25%.

echo 25 > /sys/module/zswap/parameters/max_pool_percent

To see the current compression factor of the system, run the commands below.

cd /sys/kernel/debug/zswap
perl -E "say $(cat stored_pages) * 4096 / $(cat pool_total_size)"

Distributions typically choose lz4 or lzo as the compression algorithm for zswap, however zstd is also available for more powerful devices. lzo is slightly slower than lz4 for a slightly higher compression factor. lz4 is recommended for most use cases, which could be set with the command below.

echo lz4 > /sys/module/zswap/parameters/compressor

A list of typical compression factors is available below, as well as the source of the data.

Alpine Linux

Additional repos

Gel needs the community repo to be enabled in order to function.

glibc compatibility

While not done by default, Alpine installations come bundled with a helper script configuring glibc compatibility. It may not work under every situation, but it should handle most use cases. root is required.

bash ~/gel/distro/sh/glibc.sh

openSUSE Leap

doas is not available on the platform. The regular sudo is used instead.

Security

Privilege elevation

To reduce attack surface, Gel will attempt to replace sudo with doas from OpenBSD. By default, only the root user is allowed to use the doas command.

To allow other users to execute the doas command as root, append the following directives to a new line in /etc/doas.conf.

permit keepenv <user> as root

Make sure there is a trailing new line at the end of /etc/doas.conf. If not, doas command will not work.

SSH

The SSH settings will be changed with a relatively more secure one, except for permitting password logins to prevent you from losing access with an unfinished setup.

After finishing the automated Gel setup, do the following to secure your SSH access.

1. If the password of the root user wasn't set by you, change it to a stronger one.
2. Add a custom new user, which would be used for SSH logins.
3. As the newly-created custom user, add the public keys used for SSH authorization.
4. Keep the current SSH session active, and login as the new user to ensure access to machine is not lost.
5. Allow the new user to use the doas command and verify. See the Privilege elevation section for details.
6. Add the new user to the sshuser group, added automatically by the setup script. Example: usermod -aG sshuser <user>.
7. In /etc/ssh/sshd_config, do the following.

• Set PermitRootLogin from yes to no.
• Uncomment AllowGroups and DenyGroups.
• (optional) Change the listening port from 1122 to another.

8. Restart sshd with systemctl restart sshd.

Container setup

LXC

This section documents the processes of configuring LXC on distros supported by Gel.

Installation

• Alpine: apk add lxc lxcfs lxc-download lxc-bridge
• Debian: apt install lxc lxcfs lxc-templates uidmap libpam-cgfs bridge-utils --no-install-recommends
• openSUSE: zypper in lxc
• Rocky Linux/AlmaLinux: dnf install lxc lxcfs lxc-templates
• Photon: N/A

Config files

• Container config files: /var/lib/lxc/<name>/config

Container creation from official templates

When creating containers from official templates, you'll be presented with a list of available distros, alongwith release names and CPU architectures. Visit for the full list.

To select a source image directly without the selection prompt, use the following command.

lxc-create -t download -n "<name>" -- --dist <distro> --release <release> --arch <arch>

Assign static IPv4 addresses

From Setup network bridge in lxc-net.

Create /etc/lxc/dhcp.conf. The definitions go in dhcp-host=<containerName>,<ip> format. Example below.

dhcp-host=deerHorny,10.0.3.114
dhcp-host=polakCute,10.0.3.115

If /etc/default/lxc-net exists, have the following line inside to tell lxc-net use the DHCP config before restarting the lxc-net service.

LXC_DHCP_CONFILE=/etc/lxc/dhcp.conf

Enable autostart

In the container config, have the following lines.

# Enable autostart
lxc.start.order = <startOrder> # Lower is earlier
lxc.start.auto = 1
lxc.start.delay = 4 # In seconds

Enable nested containerization

In the container config, have the following lines.

# Allow nested containerization
lxc.include = /usr/share/lxc/config/nesting.conf

Enable FUSE

In the container config, have the following lines.

# Enable FUSE
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,rw 0 0

Enable TUN

In the container config, have the following lines.

# Enable TUN
lxc.mount.entry = /dev/net dev/net none bind,create=dir
lxc.cgroup2.devices.allow = c 10:200 rwm

Limit CPU and RAM usage

From Memory Controller ・cgroup2.

In the container config, follow the example provided below.

# Limit CPU and RAM
lxc.cgroup2.memory.min = 268435456
lxc.cgroup2.memory.max = 536870912
lxc.cgroup2.cpu.max = 500000 1000000

This sets the container to...

• Use at most 512 MiB of RAM (hard limit), with 256 MiB guaranteed (hard limit).
• Allows using half of a core's worth of computing power.

Raise limits on opened files

From Proxmox ulimit hell: how to really increase open files ?

In /etc/sysctl.conf, make sure the following lines are present. Feel free to adjust the values to your needs.

fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
vm.max_map_count = 262144

In /etc/security/limits.conf, have the following lines. Feel free to adjust the values to your needs.

*     soft  nofile  1048576 unset
*     hard  nofile  1048576 unset
root  soft  nofile  1048576 unset
root  hard  nofile  1048576 unset
*     soft  memlock 1048576 unset
*     hard  memlock 1048576 unset

In the container config, have the following lines. Feel free to adjust the values to your needs.

# Raise limits on opened files
lxc.prlimit.nofile = 1048576

Inside the container, have the following lines in /etc/security/limits.conf. Feel free to adjust the values to your needs.

*     soft  nofile  1048576 unset
*     hard  nofile  1048576 unset
root  soft  nofile  1048576 unset
root  hard  nofile  1048576 unset
*     soft  memlock 1048576 unset
*     hard  memlock 1048576 unset

Reboot the host and the container(s) to apply the changes.

Manual unprivileged container setup

Extended from this blog post.

The containers configured these way are unprivileged, however they are owned by root, this is due to the problems surrounding unprivileged containers when owned by unprevileged users.

You can create the container before or after assigning subordinate IDs manually, but it must be done before modifying the container's configuration file. All commands in this section assume root privilege unless told explicitly otherwise.

Select and map subordinate IDs

Subordinate IDs permit mapping a range of IDs to a user, allowing the container to run unprivileged without the typical downsides. To avoid conflicts, it's advised to reserve a relatively large gap between different unprivileged containers in multiples of 65536, the minimum required amount of subordinate IDs for running unprivileged containers of any kind.

You'll be editing /etc/subuid for user IDs, and /etc/subgid for group IDs. Both files follow the same scheme: <username>:<startID>:<idCount>. For example, hornydeer:2097152:65536 maps IDs from 2097152 to 2162687 to user hornydeer, 65536 IDs in total. Comments are not allowed there.

As an example, we're setting the start UID and start GID to 1148576, and allocate 65536 IDs for use by the container. If you intend to have an LXC container act as a container host, you may need to scale up the count of IDs. Write the following line to both /etc/subuid and /etc/subgid.

root:1148576:65536

If you're going to run unprivileged containers inside the target unprivileged LXC, below is an example reserving enough subordinate IDs for use.

root:1148576:262144

Apply mapped IDs in configuration

To apply the mapped IDs, head to /var/lib/lxc/<containerName> and modify the config file. According to the containerized distro chosen, there may be seperate user namespace profiles, so switch to those if you encounter problems.

# Remapped user and group IDs
lxc.include = /usr/share/lxc/config/userns.conf
lxc.idmap = u 0 1148576 65536
lxc.idmap = g 0 1148576 65536

If you've chosen to use the larger ID space for unprivileged containers above, below is the corresponding example.

# Remapped user and group IDs
lxc.include = /usr/share/lxc/config/userns.conf
lxc.idmap = u 0 1148576 262144
lxc.idmap = g 0 1148576 262144

Change owner of the container root

Before the LXC container could be started, the owner of its root folder must be set as the beginning subordinate ID, 1148576 in the case of the example. Run the following command.

chown -R 1148576:1148576 /var/lib/lxc/<containerName>/rootfs

Also ensuring the container itself can access its own filesystem for good measure.

chmod 755 /var/lib/lxc # Most distros already has this as default
chmod 755 /var/lib/lxc/<containerName>
chmod 755 /var/lib/lxc/<containerName>/rootfs
chmod 640 /var/lib/lxc/<containerName>/config

nftables

The default config for nftables looks like this.

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
	chain input {
		type filter hook input priority filter;
	}
	chain forward {
		type filter hook forward priority filter;
	}
	chain output {
		type filter hook output priority filter;
	}
}

• It's possible to match multiple ports at the same time. Instead of specifying a single port number (e.g. 443), use curly braces: {443, 8443}. Ranges can also be specified: 1024-2047.
• If a certain rule only applies to traffic originating from certain interfaces, prefix the rule with iif <interface>. Can be a single interface (e.g. iif "eth0") or multiple (e.g. iif {"eth0", "ens15"}).

Transparent service exposure

From nftables: forwarding without masquerading, Quick reference: nftables in 10 minutes.

Because the LXC host is the network gateway of all LXC containers, service exposure without masquerading is entirely possible, allowing services inside LXC slices to obtain the actual IP addresses. Add the block below to begin specifying rules for service exposure.

If you want to expose services on both IPv4 and IPv6, rules will need to be duplicated. It's also important to note that containers must have the respective IP version available, for it to be exposed transparently. LXC 6.0.0 and newer has IPv6 addresses assigned automatically, while 4.0.0 and newer can have IPv6 manually configured. Only LXC 5.0.0 and newer supports IPv6 connectivity behind NAT.

table ip nat {
	chain prerouting {
		type nat hook prerouting priority filter;
		# Insert new rules for IPv4 here
	}
}
table ip6 nat {
	chain prerouting {
		type nat hook prerouting priority filter;
		# Insert new rules for IPv6 here
	}
}

Let's say we want to expose 10.0.3.2:443 for anyone on the Internet to access on port 443.

tcp dport 443 dnat to 10.0.3.2

If the port numbers are not the same, the port will need to be overriden.

tcp dport 443 dnat to 10.0.3.2:8443

Or multiple ports are to be exposed without overriding the port.

tcp dport {443, 8443} dnat to 10.0.3.2
tcp dport 512-1023 dnat to 10.0.3.2

Or only expose access to (a) certain interface(s).

iif "eth0" tcp dport 443 dnat to 10.0.3.2
iif {"eth0", "vlan0"} tcp dport 443 dnat to 10.0.3.2

An example of a rule with similar use under IPv6.

iif "he-ipv6" tcp dport {80, 443} dnat to [fc11:4514:1919:810::ff:fe00:2]

Flush your rulesets with the command below, so LXC slices will still have connectivity via NAT after flushing.

nft -f /etc/nftables.conf; systemctl restart lxc-net

Network access restriction - IP-based

Inspired by How to restrict network access of LXC container.

Notice

nftable-based network access control is still under investigation. Problems are expected to rise.

If fine-grained access control like destination-matching (e.g. domain) is desired, use EEP with transparent proxy on the host instead.

Since the current nftables approach requires static IPs to be assigned first, but there is no way found to have IPv6 addresses assigned statically, IPv6 access might need to be disabled for the container.

The inet filter forward section is where network access of individual containers is filtered.

If whitelisted network access is desired, add a rule in the scheme shown below to the end of the section for that specific container.

iif "lxcbr0" ip saddr 10.0.3.2 drop;

Then add allowed access ranges before the final drop to grant access to specific addresses. If problems occur with transparent service exposure, they will need to be made exempt.

iif "lxcbr0" ip saddr 10.0.3.2 ip daddr 10.0.3.0-10.0.3.255 accept;

Or if network access isn't whitelisted, and access to certain ranges are to be blocked, add a rule in the scheme shown below.

iif "lxcbr0" ip saddr 10.0.3.2 ip daddr 10.0.3.2-10.0.3.255 drop;

Podman

This section documents the processes of setting up Podman on distros supported by Gel. To get Podman functioning, fuse and tun support has to be present.

If you're running Podman inside an (unprivileged) LXC container, make sure the steps listed below have all been applied to the host LXC container, all of which could be found above.

• Assign a larger ID space
• Enable FUSE
• Enable nested containerization
• Enable TUN
• Raise limits on opened files

Installation

Warning

• Certain distros (e.g. Debian) may not have a functioning version of crun. Install crun from Nixpkgs when such errors are encountered.
• A few distros like Photon do not have podman-compose bundled.
• If you encounter warnings regarding / not being shared, fix temporarily with mount --make-rshared /. Read Alpine Wiki for further info.

• Alpine: apk add podman podman-compose
• Debian: apt install podman podman-compose
• openSUSE: zypper in podman podman-compose
• Rocky Linux/AlmaLinux: dnf install podman podman-compose
• Photon: tdnf install podman

After installation, run a "Hello World" container to ensure everything works correctly.

podman run --rm hello-world

If problems occur, below is an example command for debugging.

podman run --security-opt="seccomp=unconfined" --log-level=debug --rm hello-world

Manual subordinate ID assign

Note Distros may already have this section configured automatically. Only follow this section when you encounter problems.

Explanations about subordinate IDs are available in previous sections. If you encounter Podman complaining about IDs, below is an example inside unprivileged LXC containers to apply in both /etc/subuid and /etc/subgid.

<username>:65536:131072

Run podman system migrate whenever the assigned subordinate ID space changes.

Octavia

Octavia is an event-driven multi-standard MIDI state-tracking library.

Features

• Free, libre and open-source, under GNU LGPL v3.0.
• Behaves like a real MIDI module.
• Developed with Firefox and an open Web in mind.
• Supports 8 ports, 128 channels, 512-voice polyphony maximum.
• Built-in support of several standards, multiple plug-in cards, and tons of devices.
• Tells when MIDI programming errors are spotted, reducing chances of faulty programming.
• Available in JavaScript (browser and Deno).
• No modification required to run in Tor Browser, Bromite and LibreWolf.
• Wide support of bank mapping and bitmaps via midi-db.

Dev Talks

We're now hosting a new place to handle development talks! If you don't have a GitHub account, or just prefer to report bugs or give suggestions in a more casual way, feel free to chat with us with links below!

• Fediverse (Mastodon): @[email protected]
• Telegram channel: @ltgc_t
• Telegram group chat: Click to join DTM Hub

Further documentation

Demo usage

• Visualizers
• Middleware

Support coverage

• Implementation sheet
• SysEx instructions sheet
• Targets
• Standard visualizers

Implementation

• Basic messages
• Control changes
• RPN/NRPN values
• Audio Effects

SysEx documentation

• Mutual instructions
• Roland MT-32
• KORG 05/X5/N5

API documentation

• state.mjs: the core processing unit
• basic.mjs: the basis for visualizers
• bridge.mjs: the basis for middleware
• disp.mjs: ready-made example visualizers
• middle.mjs: ready-made middleware
• micc.mjs: streaming assembler and disassembler

Visualizer

Warning

This section covers how to use the visualizers bundled as public demos, as such it does not reflect the behaviour of the respective modules themselves.

Things to notice

Rendering

• Visualizers have strict render resolution constraints. For the best results, it is advised to only render while conforming to those resolution constraints.
• Visualizers are all designed to render at 50 FPS. Some visualizers may offer custom framerates, however it is advised to only render at a framerate less than or equal to the capture framerate. Visualizers offered by Octavia, especially ones displaying individual notes, feature an algorithm guaranteeing notes to be displayed each frame. As such, a framerate too high will cause visualizers fail to show display immediate notes.

Usage

Input selectors

To begin using visualizers, you must select how you want to accept events. There are several options available.

Load MIDI files

You can load standard MIDI files (.mid or .kar), load SysEx blobs (.syx), load voice remapping tables (.mdat) or custom voice bank files with this action. An audio track must be loaded with the MIDI file.

Click on the "MIDI" button to load respective files. For multi-port MIDI files, port declarations via standard meta events and XGworks port assign events are all supported. Refer to multi-port extensions for details.

Load synced audio tracks

You can load an audio file synced to the accompanying MIDI file with this action. A MIDI file must be loaded with the audio file.

Click on the "audio" button to load synced audio tracks.

Receive from the shared line-in bus

When active, the visualizer will receive events from the shared line-in bus. Events received are directly emitted from the middleware without any modifications. Read MIDI I/O routing to acknowledge how to configure how MIDI events are received.

Click on the "line in" button to begin receiving from the bus. The button may appear as just "line" in some visualizers.

Receive from the shared line-out bus

Same as "line-in", but the events received will be processed by the Octavia core present on the middleware panel page. As such, events could've been modified if polyfilling or real-time translation has been enabled. Read MIDI I/O routing to acknowledge how to configure how MIDI events are received.

Click on the "line out" button to begin receiving from the bus. The button may not be present in certain visualizers.

Mode selectors

Octavia supports a range of modes ensuring correct state handling. Upon mode switches, the corresponding mode selector will be highlighted. Clicking on the mode selectors will change the global fallback mode of the core respectively.

Visualizer-specific options

Some visualizers come with additional options for configuring behaviour and look. These options are usually quite self-explanatory, but explanations for certain options considered obscure are also available below.

Demo selector

Octavia have two demo selection systems used by various public demo visualizers: letter ID and double selection.

Letter ID

This is the simplest selection system used. Demos are indexed by a unique letter from different writing systems. Simply hitting on one of the letters loads the respective demo, while depending on the visualizer, hovering over the letters may either expose the file name or some detailed info.

Double selection

Demos are grouped by collections. To select a specific demo, first select the desired collection, then select the desired file. Pressing the "load" button will load the demo, while long presses or alt clicks will load and play the demo.

Explore

Toggle authentic mode ("true mode")

To allow visualizers function normally after possible legal strikes, most visualizers render under the "libre" mode. In libre mode, assets used are replaced with libre counter parts as much as possible, with famous examples like the libre 5 by 7 font from SynPix.

However, it is also possible for certain visualizers to try resemble the real screens as much as possible under a specific rendering mode. This mode is dubbed "true mode" or "authentic mode", and can be accessed by appending #trueMode to the URLs of visualizers, or by pressing T.

Changing channels/parts

Single-focus visualizers can only display information from one channel at a time. Focused channel can be changed by clicking on the vertical edges (left and right edges) of such visualizers, or scrolling on such visualizers.

For production purposes however, it's recommended to embed channel switch SysEx messages among the events instead. Most, if not all Octavia visualizers, support the Yamaha MU channel switch SysEx message.

Changing display modes

Certain visualizers offer various rendering modes. You can switch between modes by clicking on the top or the bottom edges of these visualizers.

Middleware

Audio Effects

Audio effects

Octavia supports tracking a range of audio effects applied on supported targets. For maximum compatibility, Octavia has seven available slots reserved for effect sends, which correspond to reverb, chorus, variation and four insertions in order.

Each slot isn't dedicated to what that slot is primarily used for, but rather allocated and controlled by the CC registers they are assigned to by default (cc91, cc93, cc94, cc16-19). For example, the variation slot (cc94) is taken away by delay effects when in GS mode, while the reverb and chorus slot could be taken away by any effect desired in X5DR or NS5R mode.

Due to varied setups, each effect also isn't just bound to the CC registers they are assigned to. They can also listen on other CC registers, or even multiple if they wish.

Comparison table

Singular effect

Reverb

Chorus

Delay

Miscellaneous

Dual effect

Mutual

• X: Yamaha XG
• G: Roland GS
• A: KORG AI²

RPN/NRPN values

RPN/NRPN values

Voice data maps

Types

ID maps

TSV files mapping MSB, PRG and LSB into voices.

• MSB: cc0 (bank MSB) value of the voice.
• PRG: PC (program change) value of the voice.
• LSB: cc32 (bank LSB) value of the voice.
• NME: 8-character ID of the voice.
• ELC: Element count (minus one) of the voice.
  • When set to 0, the voice takes up one element. 1 for two, 2 for three, the same goes on.
  • When set to 16, the voice is treated as a drum kit.
  • When set to 17, the voice is treated as a voice menu.
  • Any value above is not defined.
• DRM: Drum map ID used. Can also be used by voice menus.
• VXP: Voice property map ID used.

Drum maps

Custom script files defining properties of drum kits and their voices.

• drm defines the current drum map ID to write to.
  • e.g. drm xgStdKit switches to xgStdKit.
• dcp copies parameters from a set drum kit.
  • e.g. dcp xgStdKit copies parameters from xgStdKit to the current drum kit.
• nno defines the current drum note number to write to.
  • e.g. nno 39 switches to 39.
• npw writes multiple parameters to a single drum note.
  • npw <note> <exclusiveGroup> <elementCount>
• neg sets the exclusive group (aka. alternate assign) of the current note number. Defaults to 0 (disabled).
• nec sets the note element count of the current note number. Defaults to 1.

Voice property maps

TSV files describing certain properties of individual voices.

• VXP: Voice property ID

VL properties

• BNS: VL Breath Noise Source
  • 0: None
  • 1: Modulation wheel
  • 2: Blow strength
  • 4: Throat formant
• BNM: VL Breath Noise Amplitude from Mod Wheel
  • 0: -1
  • 64: 0
  • 127: 1
• BNB: VL Breath Noise Amplitude from Blow Strength
• BNT: VL Breath Noise Amplitude from Throat Formant
• BNV: VL Breath Noise Value Floor
  • -128: -1
  • 0: 0
  • 127: 1
• BND: VL Breath Noise Blow Strength Decay Duration
• BNL: VL Breath Noise Blow Strength Decay Level
• BAD: VL Breath Attack Duration, specified in milliseconds
• BAL: VL Breath Attack Level
  • 0: -64 (from a very wide triangle)
  • 64: 0 (no attack EG)
  • 127: 63 (from a very narrow triangle)
• BDD: VL Breath Decay Duration, specified in milliseconds
• BDL: VL Breath Decay Level
  • 0: Decay to 0 (pluck)
  • 64: Decay to 50.4%
  • 127: No decay
• TOM: VL Throat Formant Oscillation from Modulation
  • 0: None
  • 127: Maximum
• TOS: VL Throat Formant Oscillation Speed, specified in milliseconds when a cycle completes.
• TOT: VL Throat Formant Oscillation Type
  • 0: None
  • 1: Single oscillator
  • 2: Double oscillator (single oscillator modulated by an additional LFO)
• TFS: VL Throat Formant Source
  • 0: Constant until set
  • 1: cc1 (Modulation wheel)
  • ...
  • 13: cc13 (default VL throat formant source)
  • ...
  • 32: Always constant
• TFV: VL Throat Formant Value, the fixed default value for certain instruments.

Implementation Table

MIDI Implementation Chart

SysEx Instructions

For a list of supported modes, refer to state.mjs internal mode IDs.

Supported SysEx Instructions

• ✓: Supported
• -: Partially supported
• ✕: Not supported
• ?: Unknown
• (blank): N/A

Mutual instructions

1. Support in GS is called "frame draw", and with multi-page support.
2. Called "letter display" in XG, and "text insert" in GS.
3. GS only has "delay" effect occupying the space of variation setup.

Device-specific instructions

Roland MT-32

• Temporary Patch Setup
• Temporary Drum Setup
• Temporary Timbre Setup
• Device Patch Setup
• Device Timbre Setup
• Patch Memory Write
• Timbre Memory Write
• System

Yamaha MU1000

• A/D Part Setup
• A/D Mono/Stereo
• System

Yamaha PLG-100SG

• Master Setup
• Part Setup
• PhoneSEQ Setup
• Lyrics Information Setup

Yamaha PLG-150DX

• Master Setup
• Part Setup
• DX Voice Param
• DX Voice Additional Param

Yamaha PLG-150VL

• Master Setup
• Current Voice Parameters
• Part Setup

Roland SC-88

• Single/dual Mode

KORG X5D

• All Program Dump
• All Combi Dump
• Extended Multi Dump

KORG NS5R

• Mode Switch
• All Program Dump
• All Combi Dump
• Extended Multi Dump

Targets

Supported targets

General support table

The following list of targets have their support by Octavia status presented in a table. A target can be a model, a plugin board, a lineup, or a standard.

A supported standard may also have a list of specific target models listed.

For specific SysEx support range, refer to Supported SysEx Instructions. For a list of supported modes, refer to state.mjs internal mode IDs.

1. Octavia implements XG level 3.0 or later, and XG version 2.0 or later.

Custom sound banks

• Voice name maps (.mdat)
• SysEx bank dumps (.syx)
  • KORG X5 (05R/W)
  • KORG X5D (X5DR)
  • KORG NS5R (NX5R)
  • KORG Trinity
  • Yamaha PLG-AN (AN200)
• Yamaha S90 ES Voice Editor (.s7e)
• Yamaha Motif ES Voice Editor (.w7e)
• Korg Program/Combi/Global (.pcg)
  • Korg KROSS 2

Specific targets

Roland MT-32

Roland GS

Yamaha TG

Yamaha XG

Octavia standard visualizers

Octavia comes with a set of modular standard visualizers with each release.

For a guide on how to use the public demos, refer to demo visualizer usage.

Visualizers

Octavia Cambiare

• Lead developer: Lumière Élevé

The most feature-rich visualizer built on top of Octavia available. Inspired by MegaMID, vanBasco Karaoke Player and TiMidity++, Octavia Cambiare is designed from the ground up, and seeks to entirely replace what inspired it.

Octavia MU

• Lead developer: Lumière Élevé

A recreation of the screens on the Yamaha MU line up. Bitmap is collected with collective effort.

The font used for labels is Public Sans, licensed under SIL OFL.

Bitmap contributors

Special thanks to all of the people who had contributed the MU bitmap! If you see your name missing here, please contact us for proper credits.

• MIDIMan
• Lumière Élevé
• GFHK-SDGM
• MJG0117

Octavia SC

• Lead developer: Lumière Élevé

A recreation of the screens on Roland SC-55, Roland SC-88 and Roland SC-88 Pro.

The font used for labels is Work Sans, licensed under SIL OFL.

Octavia NS5R

• Lead developer: Lumière Élevé

A recreation of the screens on KORG NS5R, with help from JayB.

The font used for labels is Jost, licensed under SIL OFL.

Octavia QY

• Lead developer: Lumière Élevé

A recreation of the screens on the Yamaha QY line up, with help from JayB.

Octavia PSR

• Lead developer: GFHK-SDGM

A recreation of the screens on Yamaha PSR.

Octavia SC-8850

• Lead developer: Lumière Élevé

A recreation of the screen on Roland SC-8850.

Bitmap contributors

Special thanks to all of the people who had contributed the SC-8850 bitmap!

• SSQMinky
• MasteredRealm
• Thorndust
• GFHK-SDGM
• Lumière Élevé

Octavia MU15

• Lead developer: Lumière Élevé

A recreation of the screen on Yamaha MU15.

Support and specification table

Voice details

• N: Name only
• P: With program number
• B: With bank number
• F: With full bank numbers

Current mode

• B: Bitmap prompt
• L: Letter prompt
• C: Colour change
• D: Dedicated slot

Additional notes

1. Planned feature, not yet implemented.
2. Octavia NS5R can render voice names with at most 12 characters under certain conditions.
3. Except for Octavia MU, Octavia PSR and Octavia MU15, the intensity of pixel blur is customizable.

state.mjs API

All constants and interfaces documented here are guaranteed to work, and very likely not subject to further changes.

Constants

MIDI modes

Octavia is compatible with a range of modes on MIDI synthesizers. A list of supported modes to their respective keys is available below.

• ?: The default "nothing" mode. Octavia will try to detect the correct mode.
• gm: General MIDI mode.
• gs: Roland GS mode. GS is short for General Sound.
• xg: Yamaha XG mode. Compatible with TG-100 and TG-300. XG is short for eXtended General.
• sc: Roland GS mode, but with mode 1 or mode 2 set. Specific to Roland SoundCanvas SC-88 and up.
• g2: General MIDI Level 2 mode.
• sd: Roland SD mode. SD is used for Roland's Studio Canvas lineup.
• ns5r: KORG NS5R mode. Compatible with NX5R, and has limited compatibility with KORG N1R and N5.
• x5d: KORG X5D(R) mode. Compatible with AG-10.
• 05rw: KORG 05R/W and KORG X5 mode. Compatible with AG-10.
• k11: Kawai GMega or K11 mode.
• sg: Akai SG mode.
• krs: KORG KROSS 2 mode.
• s90es: Yamaha S90 ES mode.
• motif: Yamaha Motif ES mode.
• mt32: Roland MT-32 mode.
• doc: Yamaha DOC mode. DOC is short for Disk Orchestra Collection.
• qy10: Yamaha QY10 native mode.
• qy20: Yamaha QY20 native mode.

MIDI event types

• 8: Note off
• 9: Note on
• 10: Note aftertouch, a.k.a. polyphonic aftertouch
• 11: Channel controller change
• 12: Channel program change
• 13: Channel aftertouch
• 14: Channel pitch bend
• 15: System exclusive message

allocated

ccToPos

Interfaces

OctaviaDevice

General MIDI Extended

GME, short for General MIDI Extended, is a standard based off General MIDI level 1, which emphasizes on slightly extending the ability of General MIDI while not being too demanding. General MIDI Extended can be considered as a summarization of the common functionality among all GM-compatible synthesizers, and should be the baseline of all contemporary General MIDI-compatible synthesizers.

Requirements

Any GME-compliant synthesizer must adhere all requirements listed below.

• Has at least 24 oscillators available simultaneously (polyphony), which must either be...
  • At least 24 fully dynamically allocated oscillators for both melodic and percussion sounds.
  • At least 16 dynamically allocated oscillators for melodic sounds, 8 for percussion sounds.
• Supports all 16 MIDI channels, which...
  • Can play arbitrary number of voices within the polyphony limit.
  • Can change to a different instrument.
  • Channel 10 is set to percussion by default.
• Correctly responds to listed required events, CC and RPN.
• Has all voices listed in the required voice list.
• Has drum kits following the required drum note mapping.
• Correctly responds to required System Exclusive messages.

Voice list

Drum voices

Drum kits will reside on MSB 124, and all level 1 kits will also reside on MSB 120.

The above portion is not yet decided.

Melodic voices

Piano

Chromatic percussion

1. Some implementations may swap Dulcimer out with Santur.

Organ

Guitar

Bass

Solo strings/orchestra

Ensemble strings/orchestra

Brass

Reed

Pipe

Synth lead

Synth pad

1. Or "Fantasia Pad".

Synth melodic sound effects

Ethnic

Percussive

Sound effects

1. Can be optionally extended as a menu voice, with examples including Stadium!!! in KORG AI² lineup.

Drum note mapping

• Regular: Required.
• Italic: Optional. If not defined, the note will behave the same as in Standard Kit.
• ←: Same as Standard Kit.

Events, CC and RPN

Events

Assembly

Except System Exclusive messages, the first byte of each MIDI event defines the event type and target channel. The first four bits define the event type, with the last four bits defining the target channel.

Types

System message types

Types

Control Changes

Mapping

Data