Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1777 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.2284. Min Act: -0.0000

Data Index: 310683 (C4 (Web Text))

Max Activating Token Index: 484

Text #1

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.1254. Min Act: -0.0000

Data Index: 1277654 (C4 (Web Text))

Max Activating Token Index: 212

Text #2

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.1614. Min Act: -0.0000

Data Index: 1106330 (C4 (Web Text))

Max Activating Token Index: 358

Text #3

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.1651. Min Act: -0.0000

Data Index: 685994 (C4 (Web Text))

Max Activating Token Index: 542

Text #4

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0614. Min Act: -0.0000

Data Index: 957227 (C4 (Web Text))

Max Activating Token Index: 149

Text #5

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0649. Min Act: -0.0000

Data Index: 600389 (C4 (Web Text))

Max Activating Token Index: 696

Text #6

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0506. Min Act: -0.0000

Data Index: 885849 (C4 (Web Text))

Max Activating Token Index: 137

Text #7

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0173. Min Act: -0.0000

Data Index: 307727 (C4 (Web Text))

Max Activating Token Index: 452

Text #8

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0038. Min Act: -0.0000

Data Index: 292575 (C4 (Web Text))

Max Activating Token Index: 471

Text #9

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9557. Min Act: -0.0000

Data Index: 327198 (C4 (Web Text))

Max Activating Token Index: 600

Text #10

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9552. Min Act: -0.0000

Data Index: 797412 (C4 (Web Text))

Max Activating Token Index: 758

Text #11

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0334. Min Act: -0.0000

Data Index: 417920 (C4 (Web Text))

Max Activating Token Index: 485

Text #12

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9911. Min Act: -0.0000

Data Index: 816506 (C4 (Web Text))

Max Activating Token Index: 315

Text #13

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9834. Min Act: -0.0000

Data Index: 818756 (C4 (Web Text))

Max Activating Token Index: 624

Text #14

Max Range: 1.2284. Min Range: -1.2284

Max Act: 1.0064. Min Act: -0.0000

Data Index: 866853 (C4 (Web Text))

Max Activating Token Index: 847

Text #15

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9742. Min Act: -0.0000

Data Index: 648504 (C4 (Web Text))

Max Activating Token Index: 973

Text #16

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9690. Min Act: -0.0000

Data Index: 352680 (C4 (Web Text))

Max Activating Token Index: 552

Text #17

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9684. Min Act: -0.0000

Data Index: 818032 (C4 (Web Text))

Max Activating Token Index: 508

Text #18

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9661. Min Act: -0.0000

Data Index: 1038077 (C4 (Web Text))

Max Activating Token Index: 900

Text #19

Max Range: 1.2284. Min Range: -1.2284

Max Act: 0.9363. Min Act: -0.0000

Data Index: 615245 (C4 (Web Text))

Max Activating Token Index: 437