Cellulose – a tool to improve inference performance of ML models

Hacker News

Share card

Cellulose – a tool to improve inference performance of ML models

Hey HN! It’s Zheng here. I’m the founder of Cellulose ( https://www.cellulose.ai ). Cellulose is a tool that helps ML engineers understand, fine tune, and improve the inference performance of their ONNX models. With Cellulose, they can eventually resolve these issues in just hours, not weeks. Preparing ML models for production is a very manual and time consuming process. Unfortunately, it is also a necessary step for ML inference cost savings, sometimes even a hard requirement for certain applications like robotics and space tech. Today’s ML visualization tools are over 6 years old and lack basic features like integrating modern deep learning workflows. You’d be downloading model files locally then using a visualization tool to scroll and search for specific nodes and tensor dimensions. For example, you’ll do this twice if you’re comparing two model versions. ML researchers typically iterate on the model and then get to a “frozen”, gold release candidate before kicking off deployment related workflows. Say you use specialized hardware to run your models because that’s the most performant and cost efficient way to serve them. Unfortunately, some operators in the model could be incompatible with hardware backends like TensorRT. While there’s no shortcut but additional engineering effort to figure out a workaround or proper solution, such a setback late in the model development lifecycle is expensive for a ML team. I’ve experienced this at Cruise ( https://getcruise.com ) myself as an engineer in the Machine Learning Accelerators (MLA) team. Deploying big, bulky models onto hardware constrained environments like an AV with strict system performance limits remain a significant challenge. Friends working at various AI and robotics teams have expressed similar frustrations. Cellulose enables you to optimize and fine tune your models in a more automated fashion throughout your ML development lifecycle. We went with a product that leads with a visualizer core as so much of a ML model today is centered around the graph itself. Here’s a screenshot of a ResNet-50 model in the Cellulose dashboard: https://drive.google.com/file/d/1aZ3_fcmVVqPxxiNNcm8bkQKYsqj... Cellulose has utilities to help you copy specific values to the clipboard, just in case you’d like to run offline experimental scripts. Here’s a BatchNormalization op drawer with all its properties: https://drive.google.com/file/d/19XMY_HOwqg8ysbW4d4rqHXX5hoD... Initializer values for resnetv24_stage3_batchnorm3_gamma: https://drive.google.com/file/d/1NOwiCZbz8A2UTqDzDSnQ9WVzKVV... Export model graph as .png: https://drive.google.com/file/d/1IIOY65ZlFtc701eeMhosncSxeHd... We’re supporting Nvidia TensorRT as our first runtime. Under our Professional / Enterprise plans, we’ll annotate the TensorRT compatibility / convertibility of each node in the graph.[1] Selecting runtime type and precision options: https://drive.google.com/file/d/1Z_r68MA1HK-KVlOLA2YoPUeR0vm... TensorRT v8.6.1 compatibility badge annotations (on each op): https://drive.google.com/file/d/1L-QeZtw9gDtibJOgEdWsOm1hDNk... Supported Runtimes tab for the Reshape op: https://drive.google.com/file/d/1IS7Jio19d3WKWHh7JfrsLdzLJ7Z... We also have an exciting roadmap ( https://docs.cellulose.ai/roadmap/overview ), but more importantly, we’d like you to try it out (it’s free to start!), hear your thoughts / feedback then we’ll make sure to make those tweaks as soon as humanly possible. Feel free to sign up at http://dashboard.cellulose.ai or browse our documentation at https://docs.cellulose.ai I’ll have this tab open all day today to answer any questions! [1] - We use onnx-tensorrt for the TensorRT compatibility checks.