Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 117 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1896. Min Range: -2.1896

Max Act: 2.1896. Min Act: -0.0000

Data Index: 258878 (C4 (Web Text))

Max Activating Token Index: 578

Text #1

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.8255. Min Act: -0.0000

Data Index: 997065 (C4 (Web Text))

Max Activating Token Index: 156

Text #2

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.8075. Min Act: -0.0000

Data Index: 1063722 (C4 (Web Text))

Max Activating Token Index: 620

Text #3

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.6833. Min Act: -0.0000

Data Index: 30120 (C4 (Web Text))

Max Activating Token Index: 967

Text #4

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.7690. Min Act: -0.0000

Data Index: 915074 (C4 (Web Text))

Max Activating Token Index: 111

Text #5

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.5796. Min Act: -0.0000

Data Index: 1024662 (C4 (Web Text))

Max Activating Token Index: 312

Text #6

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.6784. Min Act: -0.0000

Data Index: 687046 (C4 (Web Text))

Max Activating Token Index: 13

Text #7

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.6289. Min Act: -0.0000

Data Index: 371758 (C4 (Web Text))

Max Activating Token Index: 136

Text #8

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.6096. Min Act: -0.0000

Data Index: 341741 (C4 (Web Text))

Max Activating Token Index: 98

Text #9

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.5765. Min Act: -0.0000

Data Index: 1313124 (C4 (Web Text))

Max Activating Token Index: 298

Text #10

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.6159. Min Act: -0.0000

Data Index: 598086 (C4 (Web Text))

Max Activating Token Index: 984

Text #11

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.4205. Min Act: -0.0000

Data Index: 190980 (C4 (Web Text))

Max Activating Token Index: 546

Text #12

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.4964. Min Act: -0.0000

Data Index: 550339 (C4 (Web Text))

Max Activating Token Index: 229

Text #13

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.4598. Min Act: -0.0000

Data Index: 899337 (C4 (Web Text))

Max Activating Token Index: 279

Text #14

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.3905. Min Act: -0.0000

Data Index: 40520 (C4 (Web Text))

Max Activating Token Index: 438

Text #15

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.3238. Min Act: -0.0000

Data Index: 34911 (C4 (Web Text))

Max Activating Token Index: 361

Text #16

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.3870. Min Act: -0.0000

Data Index: 986677 (C4 (Web Text))

Max Activating Token Index: 398

Text #17

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.3494. Min Act: -0.0000

Data Index: 1027421 (C4 (Web Text))

Max Activating Token Index: 411

Text #18

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.3495. Min Act: -0.0000

Data Index: 1265770 (C4 (Web Text))

Max Activating Token Index: 134

Text #19

Max Range: 2.1896. Min Range: -2.1896

Max Act: 1.4225. Min Act: -0.0000

Data Index: 148993 (C4 (Web Text))

Max Activating Token Index: 642