Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1715 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8717. Min Range: -2.8717

Max Act: 2.8717. Min Act: -0.0000

Data Index: 82186 (C4 (Web Text))

Max Activating Token Index: 627

Text #1

Max Range: 2.8717. Min Range: -2.8717

Max Act: 2.1594. Min Act: -0.0000

Data Index: 1023641 (C4 (Web Text))

Max Activating Token Index: 891

Text #2

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.7469. Min Act: -0.0000

Data Index: 248951 (C4 (Web Text))

Max Activating Token Index: 447

Text #3

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.6990. Min Act: -0.0000

Data Index: 710792 (C4 (Web Text))

Max Activating Token Index: 427

Text #4

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.5347. Min Act: -0.0000

Data Index: 1072431 (C4 (Web Text))

Max Activating Token Index: 977

Text #5

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.5967. Min Act: -0.0000

Data Index: 505012 (C4 (Web Text))

Max Activating Token Index: 968

Text #6

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.4902. Min Act: -0.0000

Data Index: 1121173 (C4 (Web Text))

Max Activating Token Index: 764

Text #7

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.5755. Min Act: -0.0000

Data Index: 103895 (C4 (Web Text))

Max Activating Token Index: 296

Text #8

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.4533. Min Act: -0.0000

Data Index: 700230 (C4 (Web Text))

Max Activating Token Index: 496

Text #9

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.4201. Min Act: -0.0000

Data Index: 959572 (C4 (Web Text))

Max Activating Token Index: 102

Text #10

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.3833. Min Act: -0.0000

Data Index: 905574 (C4 (Web Text))

Max Activating Token Index: 704

Text #11

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.3410. Min Act: -0.0000

Data Index: 953549 (C4 (Web Text))

Max Activating Token Index: 542

Text #12

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.3179. Min Act: -0.0000

Data Index: 80265 (C4 (Web Text))

Max Activating Token Index: 396

Text #13

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.2223. Min Act: -0.0000

Data Index: 664470 (C4 (Web Text))

Max Activating Token Index: 78

Text #14

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.2882. Min Act: -0.0000

Data Index: 1232407 (C4 (Web Text))

Max Activating Token Index: 715

Text #15

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.2828. Min Act: -0.0000

Data Index: 490213 (C4 (Web Text))

Max Activating Token Index: 408

Text #16

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.1670. Min Act: -0.0000

Data Index: 1284757 (C4 (Web Text))

Max Activating Token Index: 290

Text #17

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.2947. Min Act: -0.0000

Data Index: 36232 (C4 (Web Text))

Max Activating Token Index: 57

Text #18

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.1321. Min Act: -0.0000

Data Index: 1231734 (C4 (Web Text))

Max Activating Token Index: 442

Text #19

Max Range: 2.8717. Min Range: -2.8717

Max Act: 1.2042. Min Act: -0.0000

Data Index: 468930 (C4 (Web Text))

Max Activating Token Index: 520