Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5037 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4141. Min Range: -3.4141

Max Act: 3.2135. Min Act: -0.0000

Data Index: 969860 (C4 (Web Text))

Max Activating Token Index: 831

Text #1

Max Range: 3.4141. Min Range: -3.4141

Max Act: 3.4141. Min Act: -0.0000

Data Index: 479064 (C4 (Web Text))

Max Activating Token Index: 844

Text #2

Max Range: 3.4141. Min Range: -3.4141

Max Act: 3.2007. Min Act: -0.0000

Data Index: 415311 (C4 (Web Text))

Max Activating Token Index: 444

Text #3

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.6795. Min Act: 0.0000

Data Index: 393768 (C4 (Web Text))

Max Activating Token Index: 96

Text #4

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.7678. Min Act: -0.0000

Data Index: 23012 (C4 (Web Text))

Max Activating Token Index: 821

Text #5

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.6166. Min Act: -0.0000

Data Index: 358380 (C4 (Web Text))

Max Activating Token Index: 138

Text #6

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.6133. Min Act: -0.0000

Data Index: 472503 (C4 (Web Text))

Max Activating Token Index: 559

Text #7

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.5730. Min Act: -0.0000

Data Index: 263448 (C4 (Web Text))

Max Activating Token Index: 989

Text #8

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.3888. Min Act: -0.0000

Data Index: 1185547 (C4 (Web Text))

Max Activating Token Index: 447

Text #9

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.4925. Min Act: -0.0000

Data Index: 459798 (C4 (Web Text))

Max Activating Token Index: 633

Text #10

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.6183. Min Act: 0.0000

Data Index: 171417 (C4 (Web Text))

Max Activating Token Index: 348

Text #11

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.4812. Min Act: -0.0000

Data Index: 15935 (C4 (Web Text))

Max Activating Token Index: 858

Text #12

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.4737. Min Act: -0.0000

Data Index: 174685 (C4 (Web Text))

Max Activating Token Index: 233

Text #13

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.4605. Min Act: 0.0000

Data Index: 391635 (C4 (Web Text))

Max Activating Token Index: 51

Text #14

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.5484. Min Act: -0.0000

Data Index: 1293397 (C4 (Web Text))

Max Activating Token Index: 576

Text #15

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.2954. Min Act: 0.0001

Data Index: 1329528 (C4 (Web Text))

Max Activating Token Index: 968

Text #16

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.2639. Min Act: -0.0000

Data Index: 1326239 (C4 (Web Text))

Max Activating Token Index: 965

Text #17

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.3868. Min Act: -0.0000

Data Index: 464742 (C4 (Web Text))

Max Activating Token Index: 695

Text #18

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.3726. Min Act: -0.0000

Data Index: 379918 (C4 (Web Text))

Max Activating Token Index: 18

Text #19

Max Range: 3.4141. Min Range: -3.4141

Max Act: 2.2352. Min Act: 0.0001

Data Index: 156284 (C4 (Web Text))

Max Activating Token Index: 455