Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1107 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.4960. Min Act: -0.0000

Data Index: 506331 (C4 (Web Text))

Max Activating Token Index: 766

Text #1

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.3783. Min Act: -0.0001

Data Index: 1134063 (C4 (Web Text))

Max Activating Token Index: 243

Text #2

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.3506. Min Act: -0.0001

Data Index: 1332356 (C4 (Web Text))

Max Activating Token Index: 958

Text #3

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.2160. Min Act: -0.0000

Data Index: 1058872 (C4 (Web Text))

Max Activating Token Index: 755

Text #4

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.2544. Min Act: -0.0001

Data Index: 299339 (C4 (Web Text))

Max Activating Token Index: 481

Text #5

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.2126. Min Act: -0.0000

Data Index: 905827 (C4 (Web Text))

Max Activating Token Index: 475

Text #6

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.2093. Min Act: -0.0001

Data Index: 1113237 (C4 (Web Text))

Max Activating Token Index: 88

Text #7

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1328. Min Act: -0.0000

Data Index: 112647 (C4 (Web Text))

Max Activating Token Index: 527

Text #8

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.2551. Min Act: -0.0001

Data Index: 909102 (C4 (Web Text))

Max Activating Token Index: 388

Text #9

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1860. Min Act: -0.0001

Data Index: 898413 (C4 (Web Text))

Max Activating Token Index: 137

Text #10

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1677. Min Act: -0.0001

Data Index: 1023408 (C4 (Web Text))

Max Activating Token Index: 400

Text #11

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1387. Min Act: -0.0000

Data Index: 1170314 (C4 (Web Text))

Max Activating Token Index: 165

Text #12

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1390. Min Act: -0.0001

Data Index: 390308 (C4 (Web Text))

Max Activating Token Index: 850

Text #13

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1221. Min Act: -0.0001

Data Index: 1211271 (C4 (Web Text))

Max Activating Token Index: 41

Text #14

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1386. Min Act: -0.0000

Data Index: 31663 (C4 (Web Text))

Max Activating Token Index: 584

Text #15

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.0706. Min Act: -0.0000

Data Index: 959881 (C4 (Web Text))

Max Activating Token Index: 904

Text #16

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.0992. Min Act: -0.0000

Data Index: 325173 (C4 (Web Text))

Max Activating Token Index: 101

Text #17

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.0961. Min Act: -0.0001

Data Index: 995396 (C4 (Web Text))

Max Activating Token Index: 628

Text #18

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.1317. Min Act: -0.0001

Data Index: 737892 (C4 (Web Text))

Max Activating Token Index: 852

Text #19

Max Range: 1.4960. Min Range: -1.4960

Max Act: 1.0727. Min Act: -0.0000

Data Index: 314628 (C4 (Web Text))

Max Activating Token Index: 363