Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1117 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.1258. Min Act: -0.0000

Data Index: 911142 (C4 (Web Text))

Max Activating Token Index: 633

Text #1

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0798. Min Act: -0.0000

Data Index: 1226679 (C4 (Web Text))

Max Activating Token Index: 837

Text #2

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0889. Min Act: -0.0000

Data Index: 851545 (C4 (Web Text))

Max Activating Token Index: 63

Text #3

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0917. Min Act: -0.0000

Data Index: 8075 (C4 (Web Text))

Max Activating Token Index: 918

Text #4

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0749. Min Act: -0.0000

Data Index: 782819 (C4 (Web Text))

Max Activating Token Index: 202

Text #5

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0751. Min Act: -0.0000

Data Index: 154228 (C4 (Web Text))

Max Activating Token Index: 993

Text #6

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0873. Min Act: -0.0000

Data Index: 543592 (C4 (Web Text))

Max Activating Token Index: 707

Text #7

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0765. Min Act: -0.0000

Data Index: 539211 (C4 (Web Text))

Max Activating Token Index: 299

Text #8

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0877. Min Act: -0.0000

Data Index: 600751 (C4 (Web Text))

Max Activating Token Index: 228

Text #9

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0726. Min Act: -0.0000

Data Index: 560264 (C4 (Web Text))

Max Activating Token Index: 870

Text #10

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0774. Min Act: -0.0000

Data Index: 66678 (C4 (Web Text))

Max Activating Token Index: 437

Text #11

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0778. Min Act: -0.0000

Data Index: 728194 (C4 (Web Text))

Max Activating Token Index: 258

Text #12

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0738. Min Act: -0.0000

Data Index: 414781 (C4 (Web Text))

Max Activating Token Index: 592

Text #13

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0770. Min Act: -0.0000

Data Index: 578900 (C4 (Web Text))

Max Activating Token Index: 771

Text #14

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0737. Min Act: -0.0000

Data Index: 1090775 (C4 (Web Text))

Max Activating Token Index: 28

Text #15

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0752. Min Act: -0.0000

Data Index: 892315 (C4 (Web Text))

Max Activating Token Index: 861

Text #16

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0704. Min Act: -0.0000

Data Index: 462396 (C4 (Web Text))

Max Activating Token Index: 959

Text #17

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0672. Min Act: -0.0000

Data Index: 786260 (C4 (Web Text))

Max Activating Token Index: 708

Text #18

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0796. Min Act: -0.0000

Data Index: 504714 (C4 (Web Text))

Max Activating Token Index: 883

Text #19

Max Range: 0.1258. Min Range: -0.1258

Max Act: 0.0700. Min Act: -0.0000

Data Index: 15639 (C4 (Web Text))

Max Activating Token Index: 1001