Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2108 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.9633. Min Act: -0.0000

Data Index: 569030 (C4 (Web Text))

Max Activating Token Index: 426

Text #1

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.8919. Min Act: -0.0000

Data Index: 135801 (C4 (Web Text))

Max Activating Token Index: 440

Text #2

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.5205. Min Act: -0.0000

Data Index: 614470 (C4 (Web Text))

Max Activating Token Index: 448

Text #3

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.3706. Min Act: -0.0000

Data Index: 946958 (C4 (Web Text))

Max Activating Token Index: 677

Text #4

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.3688. Min Act: -0.0000

Data Index: 1327438 (C4 (Web Text))

Max Activating Token Index: 824

Text #5

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.5338. Min Act: -0.0000

Data Index: 366130 (C4 (Web Text))

Max Activating Token Index: 110

Text #6

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.3623. Min Act: -0.0000

Data Index: 1093438 (C4 (Web Text))

Max Activating Token Index: 162

Text #7

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.3417. Min Act: -0.0000

Data Index: 537560 (C4 (Web Text))

Max Activating Token Index: 249

Text #8

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.3836. Min Act: -0.0000

Data Index: 710271 (C4 (Web Text))

Max Activating Token Index: 144

Text #9

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.2291. Min Act: -0.0000

Data Index: 624482 (C4 (Web Text))

Max Activating Token Index: 776

Text #10

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.3688. Min Act: -0.0000

Data Index: 1080304 (C4 (Web Text))

Max Activating Token Index: 977

Text #11

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.4350. Min Act: -0.0000

Data Index: 314079 (C4 (Web Text))

Max Activating Token Index: 285

Text #12

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.2963. Min Act: -0.0000

Data Index: 563886 (C4 (Web Text))

Max Activating Token Index: 49

Text #13

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.2534. Min Act: -0.0000

Data Index: 663570 (C4 (Web Text))

Max Activating Token Index: 173

Text #14

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.2098. Min Act: -0.0000

Data Index: 849860 (C4 (Web Text))

Max Activating Token Index: 106

Text #15

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.2134. Min Act: -0.0000

Data Index: 800858 (C4 (Web Text))

Max Activating Token Index: 884

Text #16

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.1438. Min Act: -0.0000

Data Index: 734217 (C4 (Web Text))

Max Activating Token Index: 218

Text #17

Max Range: 1.9633. Min Range: -1.9633

Max Act: 0.9922. Min Act: -0.0000

Data Index: 300305 (C4 (Web Text))

Max Activating Token Index: 683

Text #18

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.1757. Min Act: -0.0000

Data Index: 1002820 (C4 (Web Text))

Max Activating Token Index: 163

Text #19

Max Range: 1.9633. Min Range: -1.9633

Max Act: 1.1737. Min Act: -0.0000

Data Index: 465249 (C4 (Web Text))

Max Activating Token Index: 315