Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5788 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1791. Min Range: -2.1791

Max Act: 2.1791. Min Act: -0.0000

Data Index: 668225 (C4 (Web Text))

Max Activating Token Index: 536

Text #1

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.7378. Min Act: -0.0000

Data Index: 1305752 (C4 (Web Text))

Max Activating Token Index: 643

Text #2

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.6759. Min Act: -0.0000

Data Index: 850935 (C4 (Web Text))

Max Activating Token Index: 57

Text #3

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.7569. Min Act: -0.0000

Data Index: 258230 (C4 (Web Text))

Max Activating Token Index: 715

Text #4

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.5303. Min Act: -0.0000

Data Index: 968855 (C4 (Web Text))

Max Activating Token Index: 390

Text #5

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.5187. Min Act: -0.0000

Data Index: 389331 (C4 (Web Text))

Max Activating Token Index: 360

Text #6

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.3748. Min Act: -0.0000

Data Index: 765271 (C4 (Web Text))

Max Activating Token Index: 166

Text #7

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.3632. Min Act: -0.0000

Data Index: 97177 (C4 (Web Text))

Max Activating Token Index: 287

Text #8

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.3626. Min Act: -0.0000

Data Index: 594489 (C4 (Web Text))

Max Activating Token Index: 444

Text #9

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.2604. Min Act: -0.0000

Data Index: 537458 (C4 (Web Text))

Max Activating Token Index: 60

Text #10

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.4088. Min Act: -0.0000

Data Index: 587941 (C4 (Web Text))

Max Activating Token Index: 480

Text #11

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.2762. Min Act: -0.0000

Data Index: 480038 (C4 (Web Text))

Max Activating Token Index: 780

Text #12

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.1485. Min Act: -0.0000

Data Index: 824289 (C4 (Web Text))

Max Activating Token Index: 289

Text #13

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.3259. Min Act: -0.0000

Data Index: 1061881 (C4 (Web Text))

Max Activating Token Index: 28

Text #14

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.1601. Min Act: -0.0000

Data Index: 1141526 (C4 (Web Text))

Max Activating Token Index: 177

Text #15

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.2572. Min Act: -0.0000

Data Index: 146222 (C4 (Web Text))

Max Activating Token Index: 283

Text #16

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.1754. Min Act: -0.0000

Data Index: 69348 (C4 (Web Text))

Max Activating Token Index: 451

Text #17

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.2199. Min Act: -0.0000

Data Index: 422567 (C4 (Web Text))

Max Activating Token Index: 478

Text #18

Max Range: 2.1791. Min Range: -2.1791

Max Act: 0.9891. Min Act: -0.0000

Data Index: 1143745 (C4 (Web Text))

Max Activating Token Index: 641

Text #19

Max Range: 2.1791. Min Range: -2.1791

Max Act: 1.1413. Min Act: -0.0000

Data Index: 1246794 (C4 (Web Text))

Max Activating Token Index: 370