Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1886 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1142. Min Range: -2.1142

Max Act: 2.0772. Min Act: -0.0000

Data Index: 467058 (C4 (Web Text))

Max Activating Token Index: 314

Text #1

Max Range: 2.1142. Min Range: -2.1142

Max Act: 2.1142. Min Act: -0.0000

Data Index: 153982 (C4 (Web Text))

Max Activating Token Index: 192

Text #2

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.8873. Min Act: -0.0000

Data Index: 29607 (C4 (Web Text))

Max Activating Token Index: 932

Text #3

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.8036. Min Act: -0.0000

Data Index: 1081097 (C4 (Web Text))

Max Activating Token Index: 77

Text #4

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.6908. Min Act: -0.0000

Data Index: 731024 (C4 (Web Text))

Max Activating Token Index: 643

Text #5

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.6387. Min Act: -0.0000

Data Index: 235741 (C4 (Web Text))

Max Activating Token Index: 622

Text #6

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5711. Min Act: -0.0000

Data Index: 1349028 (C4 (Web Text))

Max Activating Token Index: 906

Text #7

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.6376. Min Act: -0.0000

Data Index: 1050851 (C4 (Web Text))

Max Activating Token Index: 992

Text #8

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5481. Min Act: -0.0000

Data Index: 1189611 (C4 (Web Text))

Max Activating Token Index: 844

Text #9

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5995. Min Act: -0.0000

Data Index: 1335357 (C4 (Web Text))

Max Activating Token Index: 984

Text #10

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5896. Min Act: -0.0000

Data Index: 1174614 (C4 (Web Text))

Max Activating Token Index: 937

Text #11

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5785. Min Act: -0.0000

Data Index: 576522 (C4 (Web Text))

Max Activating Token Index: 640

Text #12

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5793. Min Act: -0.0000

Data Index: 843574 (C4 (Web Text))

Max Activating Token Index: 868

Text #13

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.6682. Min Act: -0.0000

Data Index: 289412 (C4 (Web Text))

Max Activating Token Index: 662

Text #14

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5583. Min Act: -0.0000

Data Index: 908149 (C4 (Web Text))

Max Activating Token Index: 968

Text #15

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5551. Min Act: -0.0000

Data Index: 115446 (C4 (Web Text))

Max Activating Token Index: 805

Text #16

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5144. Min Act: -0.0000

Data Index: 1184869 (C4 (Web Text))

Max Activating Token Index: 772

Text #17

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5442. Min Act: -0.0000

Data Index: 140442 (C4 (Web Text))

Max Activating Token Index: 909

Text #18

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.5580. Min Act: -0.0000

Data Index: 970114 (C4 (Web Text))

Max Activating Token Index: 351

Text #19

Max Range: 2.1142. Min Range: -2.1142

Max Act: 1.4967. Min Act: -0.0000

Data Index: 457318 (C4 (Web Text))

Max Activating Token Index: 946