Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1559 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.1220. Min Range: -6.1220

Max Act: 6.1220. Min Act: -0.0000

Data Index: 1126098 (C4 (Web Text))

Max Activating Token Index: 526

Text #1

Max Range: 6.1220. Min Range: -6.1220

Max Act: 3.6745. Min Act: -0.0000

Data Index: 195092 (C4 (Web Text))

Max Activating Token Index: 988

Text #2

Max Range: 6.1220. Min Range: -6.1220

Max Act: 3.1976. Min Act: -0.0000

Data Index: 345269 (C4 (Web Text))

Max Activating Token Index: 953

Text #3

Max Range: 6.1220. Min Range: -6.1220

Max Act: 3.1630. Min Act: -0.0000

Data Index: 1188076 (C4 (Web Text))

Max Activating Token Index: 555

Text #4

Max Range: 6.1220. Min Range: -6.1220

Max Act: 3.0800. Min Act: -0.0000

Data Index: 1333963 (C4 (Web Text))

Max Activating Token Index: 31

Text #5

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.9641. Min Act: -0.0000

Data Index: 490095 (C4 (Web Text))

Max Activating Token Index: 697

Text #6

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.9207. Min Act: -0.0000

Data Index: 509399 (C4 (Web Text))

Max Activating Token Index: 836

Text #7

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.6685. Min Act: -0.0000

Data Index: 376054 (C4 (Web Text))

Max Activating Token Index: 620

Text #8

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.7703. Min Act: -0.0000

Data Index: 1196425 (C4 (Web Text))

Max Activating Token Index: 787

Text #9

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.6013. Min Act: -0.0000

Data Index: 369093 (C4 (Web Text))

Max Activating Token Index: 947

Text #10

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.7813. Min Act: -0.0000

Data Index: 1170630 (C4 (Web Text))

Max Activating Token Index: 488

Text #11

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.7471. Min Act: -0.0000

Data Index: 655783 (C4 (Web Text))

Max Activating Token Index: 397

Text #12

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.4942. Min Act: -0.0000

Data Index: 245652 (C4 (Web Text))

Max Activating Token Index: 396

Text #13

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.4050. Min Act: -0.0000

Data Index: 33295 (C4 (Web Text))

Max Activating Token Index: 731

Text #14

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.2675. Min Act: -0.0000

Data Index: 476326 (C4 (Web Text))

Max Activating Token Index: 822

Text #15

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.4734. Min Act: -0.0000

Data Index: 290750 (C4 (Web Text))

Max Activating Token Index: 950

Text #16

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.4515. Min Act: -0.0000

Data Index: 838617 (C4 (Web Text))

Max Activating Token Index: 418

Text #17

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.3238. Min Act: -0.0000

Data Index: 1273018 (C4 (Web Text))

Max Activating Token Index: 357

Text #18

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.0914. Min Act: -0.0000

Data Index: 934393 (C4 (Web Text))

Max Activating Token Index: 788

Text #19

Max Range: 6.1220. Min Range: -6.1220

Max Act: 2.2665. Min Act: -0.0000

Data Index: 589923 (C4 (Web Text))

Max Activating Token Index: 273